benchturn rests on a claim about training data: that models trained on final outputs alone reach a plateau, while models trained on how people actually work, the decisions, corrections and checks along the way, keep improving. This review asks what the published literature actually supports, where that support stops, and what remains an open question.

Two kinds of supervision

In the training literature, outcome supervision means rewarding a model on whether its final answer is correct. Process supervision means rewarding or checking each intermediate step. The distinction matters because a model can arrive at a correct answer by faulty reasoning, and outcome supervision has no way to penalise that.

"Process data" in the benchturn sense is broader than the step labels used in these papers. It is the record of how a practitioner reaches a result: which options they considered, what they rejected and why, where they checked a figure against a source, and how they handled the case that did not fit the template. The published studies are the narrowest version of this idea, and the distance between them and a lawyer's or an accountant's working day should be kept in view throughout.

What the process-supervision studies found

Uesato et al. (2022) at DeepMind compared the two approaches on GSM8K, a set of grade-school maths word problems. Their result is more nuanced than it is often quoted. On final-answer accuracy, outcome-based supervision performed comparably to process-based supervision, and needed less labelled data to get there. The difference appeared in the reasoning: outcome-based methods let through solutions that reached the right answer by wrong steps, and process-based feedback was needed to bring that error rate down.

Lightman et al. (2023) at OpenAI took the comparison to the harder MATH dataset and found a clearer gap. A reward model trained with step-level human labels outperformed one trained with outcome labels, and their best model solved 78% of a representative subset of the MATH test set. They released PRM800K, roughly 800,000 step-level labels, and found that active learning, choosing which solutions to send for labelling, made the human effort go further. The two studies are not in conflict so much as at different scales and difficulties: on easier problems the outcome signal is nearly sufficient; on harder multi-step problems, supervising the steps pays off.

Both papers are about mathematics, where each step can be judged correct or incorrect by an annotator and the final answer checked automatically. That is what makes them clean experiments, and it is also what limits them.

Small amounts of good human data

A second strand of evidence concerns how much human data it takes to change a model's behaviour. Ouyang et al. (2022) fine-tuned GPT-3 on human-written demonstrations and then on human rankings of outputs, producing InstructGPT. In their human evaluations, outputs from the 1.3 billion parameter InstructGPT model were preferred to those of the 175 billion parameter GPT-3. Human feedback on what good behaviour looked like moved the model further than a hundredfold increase in size.

Zhou et al. (2023) pushed the question further. LIMA fine-tuned a 65 billion parameter model on only 1,000 carefully selected prompts and responses, with no reinforcement learning and no preference data. In human comparisons, its responses were judged equivalent to or better than GPT-4's in 43% of cases, and it did better still against other models of the time. The authors' reading is that almost all of a model's knowledge is acquired in pretraining, and that a small amount of high-quality data is enough to teach the style and format in which to use it.

Read carefully, these two papers support a narrower point than "a little data goes a long way". They show that a little data goes a long way for alignment and format. Whether the same is true for teaching a model new competence in a domain, such as the hard case in tax law, is not what they tested.

The ceiling on scraped text

The third strand is what happens when the supply of public text runs down. Villalobos et al. (2022, updated 2024) at Epoch AI estimated the stock of public human-generated text and projected demand against it. Under current trends, they expect models to be trained on datasets the size of that whole stock at some point between 2026 and 2032. The authors point to synthetic data, transfer from other domains and better data efficiency as the routes past that limit.

Muennighoff et al. (2023) asked how much of that limit can be recovered by reusing data. Across a large set of training runs, repeating data for up to about four epochs cost little compared with fresh data at the same compute budget; past that point, the returns from additional compute on repeated data fell towards zero.

Shumailov et al. (2024), in Nature, examined the synthetic-data route directly. When models are trained on content generated by earlier models without care, the tails of the original distribution disappear and the defects compound across generations, a phenomenon they call model collapse. They demonstrate it in language models, variational autoencoders and Gaussian mixture models, and argue that access to genuine human data becomes more valuable, not less, as generated content accumulates on the web.

Taken together, the three papers say something specific: the stock of scraped text is finite, reuse recovers only a few epochs of it, and replacing it with model output has a known failure mode. None of them says that process data from practitioners is the answer. They do say that the default supply is not indefinite.

Benchmarks and contamination

If scraped data is running down, one might expect progress to show up as flattening benchmark scores. The contamination literature complicates that reading. Sainz et al. (2023) argue that when benchmark test sets leak into training data, reported scores overstate a model's ability and can lead to wrong conclusions being published. Yang et al. (2023) show that the standard defence, n-gram overlap detection, is easy to bypass: rephrased or translated test items pass through it, and a 13 billion parameter model fine-tuned on such rephrasings can match GPT-4 on MMLU and HumanEval. Deng et al. (2023) approach it from the other side, testing whether models can fill in masked parts of benchmark questions; on MMLU, ChatGPT and GPT-4 reproduced missing answer options exactly in 52% and 57% of cases.

The relevance to the plateau question is indirect but real. Benchmark scores are an unreliable guide to whether models are still improving on the underlying tasks, which makes it harder, not easier, to tell whether output-trained models have levelled off. It also means that evaluation on real work, held out from the web, is worth more than another public leaderboard.

What the evidence does not show

We should be direct about the limits. The process-supervision results come from mathematics, with related work in code, where every step admits a right or wrong label. There is little published evidence, positive or negative, on process supervision in law, medicine, accounting or other professional fields where steps are matters of judgement and the correct answer is often contested. The data-efficiency results concern alignment and format, not domain competence. The data-limit results concern public text, and say nothing about the private record of professional work. And "process data from real work" in the sense benchturn means it, decisions and corrections captured in the course of an ordinary working day, has not been studied at scale by anyone. This is a gap in the literature. It is not a result in either direction.

What we take from this

In the domains where the experiments exist, the published results are consistent with the thesis: supervising the steps beats supervising the answer on hard problems, small quantities of careful human data shift behaviour more than scale does, and the scraped-text supply has a ceiling with a known failure mode past it. Whether real-work process data from practitioners behaves the same way is the open question, and it will not be settled by argument. It will be settled by labs training on such data, in domains where the outcome is contested, and reporting what happens. That is the question benchturn wants to help answer. If you are running that experiment, or want to, write to hello@benchturn.com.

References

  • Deng, C., Zhao, Y., Tang, X., Gerstein, M. and Cohan, A. (2023). Investigating Data Contamination in Modern Benchmarks for Large Language Models. https://arxiv.org/abs/2311.09783
  • Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I. and Cobbe, K. (2023). Let's Verify Step by Step. https://arxiv.org/abs/2305.20050
  • Muennighoff, N., Rush, A. M., Barak, B., Le Scao, T., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T. and Raffel, C. (2023). Scaling Data-Constrained Language Models. https://arxiv.org/abs/2305.16264
  • Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J. and Lowe, R. (2022). Training language models to follow instructions with human feedback. https://arxiv.org/abs/2203.02155
  • Sainz, O., Campos, J. A., García-Ferrero, I., Etxaniz, J., Lopez de Lacalle, O. and Agirre, E. (2023). NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. Findings of EMNLP 2023. https://arxiv.org/abs/2310.18018
  • Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R. and Gal, Y. (2024). AI models collapse when trained on recursively generated data. Nature, 631, 755-759. https://www.nature.com/articles/s41586-024-07566-y
  • Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G. and Higgins, I. (2022). Solving math word problems with process- and outcome-based feedback. https://arxiv.org/abs/2211.14275
  • Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L. and Hobbhahn, M. (2022, revised 2024). Will we run out of data? Limits of LLM scaling based on human-generated data. https://arxiv.org/abs/2211.04325
  • Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E. and Stoica, I. (2023). Rethinking Benchmark and Contamination for Language Models with Rephrased Samples. https://arxiv.org/abs/2311.04850
  • Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., Zhang, S., Ghosh, G., Lewis, M., Zettlemoyer, L. and Levy, O. (2023). LIMA: Less Is More for Alignment. https://arxiv.org/abs/2305.11206