Architectures of frontier AI: choices and tradeoffs

Revision note, October 16, 2026. This essay was substantially revised to add primary references, narrow unsupported claims and distinguish proposals from measured results. The original address and publication date are retained.

Model architecture, training data and serving constraints interact. A useful comparison separates disclosed designs from measured outcomes and leaves undisclosed choices unknown.

Start with named models

There is no basis for classifying every frontier model as a mixture of experts. The Llama 3 report describes dense models, including its 405B model. The GPT-4 report withholds further architectural details. Public comparisons should preserve that distinction rather than filling an undisclosed design with an inference presented as fact. Llama 3, GPT-4, section 2.

A mixture-of-experts design routes work through selected experts; a dense design uses a different allocation of parameters and computation. Neither label determines useful quality or serving cost on its own. Comparisons need an actual model, workload, hardware setup and measurement protocol.

The term frontier also needs a boundary. A comparison can define its sample through named evaluations, a capability threshold or a dated set of releases. Without that rule, statements about what all frontier architectures share cannot be independently checked.

Separate attention choices

Attention patterns, key/value-head sharing and implementation algorithms describe different aspects of a system. Treating them as interchangeable categories obscures what a design actually changes.

FlashAttention is an IO-aware algorithm for computing exact attention. It is not an alternative sparsity pattern. Its contribution concerns how attention is executed and how data moves through memory. FlashAttention.

Mistral 7B describes grouped-query attention and sliding-window attention. A window at one layer does not justify a blanket claim that earlier information is unavailable throughout the model: information flow also depends on the stack of layers. Effective long-context behavior still needs to be tested on the intended task. Mistral 7B.

These distinctions suggest a better comparison record: identify the attention pattern, head-sharing scheme and execution method separately, then measure the outcomes relevant to deployment.

Include hybrid designs without assuming their rank

Jamba documented a Transformer-Mamba hybrid with mixture-of-experts layers in 2024. Its existence establishes a published hybrid design, not an automatic place at the top of every capability ranking. Jamba.

A claim that such a design belongs outside a chosen frontier sample needs the same capability criteria used for other models. A claim that it is superior needs comparable measurements. Architectural novelty and demonstrated performance are separate properties.

Compare the system, not just its parameter count

For each candidate, record the disclosed architecture, training recipe where available, context protocol, task quality, latency, memory use and serving budget. Mark missing information as unknown. Keep model versions and inference settings fixed when attributing a difference to architecture.

Do not infer reciprocal training arrangements or private investment priorities from the fact that distillation or synthetic training is technically possible. Those claims need specific public evidence about the organizations and training runs involved.

Questions worth testing

A useful experiment would compare designs under an explicit resource constraint and report where the ranking changes. Does a memory advantage survive a longer context? Does a lower serving cost hold at the same task quality? Does post-training change the relative failure pattern?

These are research questions, not findings established by this essay. The contribution of a comparison is to make those questions testable while keeping the boundary between disclosure, observation and interpretation visible.