The idea in one paragraph.
Scaling an autoregressive LM to 175B parameters makes task adaptation possible purely by placing examples in the context, with no gradient updates at inference.
THE MACHINERY
In-context learning
Condition on demonstrations in the prompt; weights stay frozen.
Zero/one/few-shot settings
Vary the number of in-prompt demonstrations as the only lever.
What the paper reports.
Findings are reproduced as concise corpus summaries, with their original confidence labels. They are not independent replications.
Trains GPT-3, an autoregressive LM with 175 billion parameters, 10x more than any previous non-sparse model.
Inspect a source excerpt
we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language modelRead the passage in context on arXiv ↗
Few-shot performance improves with model scale, so the context becomes a usable control surface.
LIMITATIONS / CORPUS READING
Where to keep your judgement.
- No mechanism to act, verify, or persist anything; the context is written by the operator, never by the model.
- Few-shot gains saturate once the prompt is full - the talk notes ICL stops improving after roughly 40-50 examples.
Why it belongs in this atlas.
The first move of the whole field: change behaviour by changing the context rather than the weights. Everything later is an argument about who gets to write that context.