Every AI chatbot you have used writes one word at a time, even the fast ones. Next-Concept Modeling (NCM) asks a different question: what if a model sketched the whole answer first, then filled in the detail, in about a dozen steps?
Researchers at Stanford’s Hazy Research lab published exactly that, and it reports text generation up to 31 times faster than strong diffusion language models, with 4 to 5 times fewer training tokens.
All the results below come from the authors’ own paper, which is a preprint. I could not find an independent reproduction yet, so read them as reported, not confirmed.
Why language models are slow
A normal language model is autoregressive. It predicts one token, adds it to the text, then predicts the next one. If you want a refresher on what a model does with text, my post on how AI agents work covers the basics.
A 1,000 token answer therefore takes about 1,000 steps, one after another. The model can’t skip ahead, and it can’t go back and change an early word. That is why long answers take time, and why inference is now one of the biggest costs in AI.
So researchers have been looking for ways to generate more than one token at a time.
The diffusion alternative
The leading alternative is the diffusion language model. It starts with the whole sequence masked, meaning every position is blank. At each step it predicts all the blanks in parallel, keeps its most confident guesses and leaves the rest masked for the next round. It locks words in gradually, in no fixed order.
That sounds fast, but each step runs the whole neural network over every position, including the ones already settled. For a 1,000 token answer, every step processes all 1,000 positions, again and again. You usually need many steps for good quality, and these models often need many passes over the training data too.
In the paper, a strong diffusion baseline uses 128 generator calls on the LM1B benchmark. NCM needs about a dozen. Cutting steps normally costs quality, so NCM takes a different route.
The NCM idea: sketch first, detail later
Think of a novelist. Before writing a sentence, they already have the concept: a lighthouse keeper, a storm, a letter that arrives too late. They know how it starts, where it turns and how it ends. Then they outline the chapters, draft the scenes and only at the very end polish each sentence and pick each word. The whole story exists from the start as a rough shape, and every pass adds detail.
NCM generates text in that order. To make it work, it first learns codebooks from a lot of training text.
- A codebook is a numbered list of options, like a menu. Each option stands for a piece of information about the text, sometimes broad and sometimes fine-grained.
- A code is the number of the option that was picked.
- NCM represents text as a hierarchy of these codes. The early levels are short, a few codes capturing the broad structure. Later levels are longer and add finer detail.
The model generates one level at a time, and inside each level it predicts every code in parallel. So instead of one step per token, or dozens of full passes, you get one step per level. Depending on the task, the paper uses between 8 and 32 levels, most often around a dozen. Because the first levels are short, the early steps are cheap.
The idea is borrowed from image generation, where next-scale prediction makes a low-resolution picture first and then sharpens it. NCM brings that to text.
The word “concept” is a little misleading: here it means information in the coarse levels of a learned code hierarchy, not human-readable ideas.
How NCM works
NCM has two parts: a tokenizer that acts as a compressor, and a generator that acts as a writer. Both use the same set of levels, rough first and fine last.
Training
Training starts with the compressor. An encoder creates the codes, one set per level, and a decoder rebuilds the text from them; each level has its own codebook. The authors sometimes switch the later levels off during training, which forces the early levels to carry real information on their own. Training also compares the rebuilt text with the original token by token, so nothing important gets lost.
The writer trains second, with the compressor locked in place. A transformer called the Next-Concept Transformer learns to predict the codes for each level from the levels before it. It trains on all levels in one pass, and the authors deliberately swap some of the codes it receives for wrong ones, so it learns to cope with its own mistakes.
Generation
At generation time there is no more learning. The writer produces the codes for level one, then level two, and so on, with every code inside a level predicted at once. When the last level is done, the compressor’s decoder turns the codes back into text. Most tasks take roughly a dozen steps, one per level.
| Approach | What one step does | Steps for a long answer |
|---|---|---|
| Autoregressive LLM | Predicts the next token | About one per token |
| Diffusion language model | Refines the whole sequence | Many passes (128 in the paper’s baseline on LM1B) |
| NCM | Predicts one whole level of codes at once | About a dozen |
The reported results
The authors tested three kinds of tasks.
- Writing from scratch, with no prompt, on TinyStories (short, simple stories), LM1B (sentences from news articles) and OpenWebText (general web text).
- Translation from German into English, on the WMT14 test set.
- Summarization on XSum, where the model squeezes a BBC news article into one sentence.
Measured by wall-clock time on a single H100 GPU, and compared with the strongest diffusion baseline:
| Task | Reported result |
|---|---|
| LM1B | 21x faster, with better generative perplexity |
| OpenWebText | 10x faster, with competitive quality |
| XSum | 31x faster, at the same ROUGE-L score |
Training is cheaper too. NCM used about 4.9 times fewer training tokens on LM1B and about 4 times fewer on OpenWebText, counting both the tokenizer and the generator.
The authors also show early scaling results. Quality improves going from 97 million to 317 million to 725 million parameters, though they call these results preliminary. And because NCM is fast, you can sample many candidates in the same time budget and pick the best with minimum Bayes risk decoding, which improved translation and summarization scores.
The comparison models are research models of a similar, small size, mostly diffusion models. The benchmarks leave out the large LLMs behind popular chatbots, so we can’t yet say how NCM compares with them.
Sudoku and reasoning
There is one more test: Sudoku. The authors’ guess is that sketching the big picture first should help with problems that need planning. Sudoku fits, because every number has to agree with the others in its row, column and box.
| Difficulty | Left-to-right model | Best competitor | NCM |
|---|---|---|---|
| Easy | 14.6% | 97.7% | 99.5% |
| Medium | 5.1% | 92.8% | 96.5% |
| Hard | 1.0% | 81.3% | 86.1% |
NCM needed just 12 steps per puzzle, where most of the others used 128. It is a narrow benchmark, so don’t over-read it. But it fits the idea that sketching first helps with planning.
Try NCM yourself
The code is open source under the MIT license in the HazyResearch/ncm repo, and the checkpoints are on Hugging Face. Released models cover XSum summarization, WMT14 translation, three Sudoku difficulties, TinyStories, OpenWebText and LM1B in small, medium and large. The biggest is 725 million parameters.
Run it locally
Clone the repo, install the requirements, download the checkpoints and run evaluate.py with a task config. The repo recommends Python 3.10 or newer and a CUDA-capable PyTorch install. The repo is built for evaluation. It has no chat interface and no ready-made script for your own prompts.
Use the browser demo
If you would rather not install anything, there is a community demo on Hugging Face. It is not made by the authors. It is called Next-Concept Modeling, by mrfakename, and it runs on Hugging Face’s GPUs, so you don’t need one. It has five tabs: TinyStories, LM1B, XSum, WMT14 and Sudoku.
- TinyStories: click Write a story, then drag the Level slider. Level one is the rough first sketch and every later level adds detail. The All levels section lists the draft from every level.
- XSum: paste a news article or pick an example, click Summarise and drag the Level slider to watch the one-sentence summary take shape.
- WMT14: type a German sentence, click Translate and watch the English version sharpen from level to level.
- Sudoku: choose a difficulty, enter or load a puzzle and press Solve. The slider shows the grid filling in.
On the story and sentence tabs, the Settings menu lets you fix the seed or change the temperature, which controls how varied the writing is.
NCM vs LLMs: is this the end of the LLM?
Not yet.
- The speed claims are against diffusion models, not large production LLMs, and the biggest NCM here has 725 million parameters. Frontier chat models are far larger.
- The paper’s tests don’t cover chat, following instructions, long documents or factual accuracy, so we don’t know how NCM handles them.
- The model still generates the levels one after another, so the parallelism is within each level, not across the whole output.
- The two-part design has a risk: if the tokenizer loses detail, the generator can’t bring it back.
- These are author-reported results from a preprint, with no independent reproduction that I could find.
Don’t confuse NCM with Meta’s Large Concept Models, which predict the next sentence instead.
Recap
Most chatbots write left to right, one token at a time. NCM writes the outline first and fills in the detail level by level, in parallel. On small research benchmarks that gave it up to 31 times faster generation than diffusion models, with 4 to 5 times fewer training tokens. What we don’t know is whether it holds up at the scale of the chatbots we use every day.
The team lists further scaling, joint training of the tokenizer and generator, and adapting the number of levels to difficulty as next steps. If you try the demo, tell me what you see in the video comments.
Sources
- Paper: Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation (arXiv 2610.14162)
- Hazy Research blog post
- GitHub: HazyResearch/ncm
- Checkpoints on Hugging Face
- Community demo: mrfakename/ncm
Related posts
For another look at how models are used in code, read How AI agents work and the Jev tutorial.