The idea in one paragraph.
Put a language model behind a real text browser, let it search and quote, and train it with human feedback so its answers carry verifiable references.
THE MACHINERY
Text-based browsing action space
A fixed command vocabulary - search, click, quote, scroll - over a real browser.
Best-of-n rejection sampling
Sample several answers and pick the best under a learned reward model.
Citation-backed answers
Answers must carry quoted references collected during browsing.
What the paper reports.
Findings are reproduced as concise corpus summaries, with their original confidence labels. They are not independent replications.
The best model fine-tunes GPT-3 with behaviour cloning, then does rejection sampling against a reward model trained to predict human preferences.
Inspect a source excerpt
Our best model is obtained by fine-tuning GPT-3 using behavior cloning, and then performing rejection sampling against a reward model…Read the passage in context on arXiv ↗
Its answers are preferred by humans 56% of the time to human demonstrators, and 69% of the time to the highest-voted Reddit answer.
Models must collect references while browsing, so factual accuracy is easier to check.
LIMITATIONS / CORPUS READING
Where to keep your judgement.
- The action space is hand-designed and fixed; the model cannot add a browser command.
- Reward-model optimisation can favour answers that look well-sourced over answers that are correct.
Why it belongs in this atlas.
First rung out of the context: the model gets to change the world, but only through a vocabulary the designer wrote in advance.