Scroll to watch the architecture assemble itself, block by block. Every concept you scroll past becomes visible — and active — in the diagram.
Every input — text, image patches, audio samples — is broken into tokens, the atoms of meaning. NEXUS speaks 134 languages, but the model only knows tokens.
Each token becomes a vector in a 12,288-dimensional space. Position encodings tell the model where each token sits in the sequence — order matters.
Every token looks at every other token, in parallel, across 96 attention heads. This is the trick that lets the model reason about long contexts — and it scales gracefully.
def attention(q, k, v): scores = q @ k.T / sqrt(d_k) weights = softmax(scores) return weights @ v
Two dense layers per block sharpen the representation, projecting through a 49,152-dim hidden space. This is where most of the parameters live, and where most of the "knowledge" is stored.
The attention + FFN block repeats 96 times, with residual connections and layer normalization between each. Information flows up, gets refined, flows up again.
The final layer projects back to vocabulary space — a probability for every possible next token. We sample. We continue. A response is born, one token at a time, at 240 tokens/sec.
NEXUS is trained in four stages — each stage compounding the previous one's gains.
Trillions of tokens — books, code, scientific papers, the open web. The model learns the texture of language.
High-quality demonstrations from experts. The model learns how to be helpful, not just what humans write.
Reinforcement learning from human feedback. Humans rank outputs; the model learns the ranking. Subtle, hard, essential.
The model critiques its own outputs against a constitution of principles. Alignment without endless human labeling.
A model is only useful if it answers fast. NEXUS uses speculative decoding, paged attention, and KV-cache quantization to hit 42ms time-to-first-token and stream at 240 t/s.