How often do people correct models in the wild?

Anyone who uses a model for real work corrects it. The accountant says the election was made in the wrong year; the engineer says the patch moves a lock outside the loop. Those moments are the signal benchturn exists to route to the labs, with consent and payment. We have been claiming they are frequent without a number attached, so we went looking for one. The question here is narrower than the thesis: in ordinary, public, unpaid use of a chat model, how often does the user tell the model it got something wrong, and what do those corrections contain? We used a public corpus of real ChatGPT logs and a deliberately simple method.

Data

We used WildChat, a corpus of real ChatGPT conversations collected through a free public proxy whose users consented to logging (Zhao et al., "WildChat: 1M ChatGPT Interaction Logs in the Wild", ICLR 2024; https://huggingface.co/datasets/allenai/WildChat). The public split holds 529,428 conversations. We read it through the Hugging Face datasets-server rows API, fetching 50 pages of 100 rows at offsets spaced evenly from 0 to 529,328. No page failed. That gave 5,000 conversations, of which 2,635 are labelled English and 996 of those have at least two user turns. Only a multi-turn conversation can contain a correction, so those 996 are the study population: 879 ran on gpt-3.5-turbo, 113 on gpt-4 and 4 on a gpt-4 preview, between April and November 2023, with a median of 3 user turns.

Method

We wrote a rule-based classifier over user turns. A turn is eligible if it is the user's second or later message and follows an assistant message. The text is lower-cased and stripped of leading punctuation, then matched against two families of regular expressions. The first flags a correction: openings such as "no,", "not quite", "that's wrong", "you missed", "actually", "it should be", "that doesn't work", "not what I asked", "try again", "fix it", "but I" and "instead of", plus error signatures such as a Python traceback anywhere in the first 400 characters. The second flags steering, a refinement that does not say the model was wrong: "make it shorter", "add", "change X to Y", "more formal", "in Python instead". Correction takes precedence, so the categories do not overlap.

The obvious trap is a user pasting their own text behind an editing verb: "Correct this sentence: ..." is a task, not a complaint, and an early version counted a dozen of them. The final rules skip a turn as pasted material when it contains "the following" or "below", or a colon, semicolon or quotation mark in its first 40 characters, unless an error signature is present. Questions and a bare "continue" are not counted.

To estimate precision we drew a seeded random sample of 60 flagged corrections and 40 flagged steering turns and judged them by hand against the preceding assistant turn. A correction counted as true if the turn says the previous answer was wrong, did not work, or did not do what was asked; a pasted error counted when it followed code or steps the model had given. The rules were refined over two earlier rounds on same-seed samples, so the estimate is not fully independent of the tuning. Recall was not measured.

Results

Of the 996 multi-turn English conversations, 83 (8.3%) contain at least one flagged correction, 119 (11.9%) contain at least one steering turn, and 188 (18.9%) contain either. At the turn level, 112 of the 3,216 eligible user turns (3.5%) are corrections and 203 (6.3%) are steering. The correction rate barely moves with position: 3.4% at the second user turn, 4.1% at the third, 3.7% at the fourth and 3.2% at the fifth and later.

Manual precision was 49 of 60 for corrections (0.817) and 29 of 40 for steering (0.725). Multiplying the raw shares by those figures gives a precision-adjusted 6.8% of conversations with a correction and 8.7% with steering, or 2.8% of eligible turns being corrections. Unknown recall pushes the other way, so we read 8.3% as a rough central figure rather than a bound.

The domain split is a crude keyword heuristic over the whole conversation (a code fence, "function", "error", "python" or "javascript" means code; "essay", "story", "email", "rewrite" or "paragraph" means writing). It over-assigns to code, since "error" alone is enough, but the gap is large.

Per eligible turn the correction rate is 5.4% in code, 2.4% in writing and 1.9% elsewhere. People correct models most where the output can be run.

What the corrections look like

Ten flagged turns, lightly edited to remove anything identifying. In code, a user writes "I tried to implement it but it doesn't work on my page" after the model insisted a snippet had no mistake. Another pastes "ParserError: Expected 37 fields in line 1429, saw 52" and adds that the source files keep their data in separate cells. A third says "I said in C, not in Lisp." A fourth, using simulation software, writes "you are wrong, the first value is the initial increment and second is total time". A fifth catches a contradiction: "but you said y = -1/3 f(x) will face a vertical compression, isn't -1/3 smaller than 1?" A sixth reports "same wrong result, still all output null".

Outside code the corrections are about frame and form. "No, the year is actually 5073" restores a role-play premise. "But it is not the Cucuteni-Trypillia proto-writing, it is Sumerian script" asserts a different attribution, and the user may well be wrong. A medical student says "No, I want your answer to be like this" and supplies a compressed revision format. A children's story comes back with "Redo. The balloon character pops but does not die."

What could a lab learn? The parser error and the contradiction are the best cases: the failure is stated, and one is checkable from the conversation alone. The simulation correction is the most valuable in kind, a sentence of expert knowledge the model lacked, and the least usable, because nothing in the log says whether the user is right. Most of the rest say an answer failed without saying why, and the input needed to reproduce the failure never appears. The corrections are real, frequent enough to matter, and thin.

Limitations

The classifier is a list of patterns, with precision estimated on a small hand-judged sample that overlaps with the tuning samples and recall unknown; polite or indirect pushback is invisible to it. WildChat users are self-selected, having chosen a free proxy and consented to logging, and many were testing rather than working. The model is ChatGPT as served in 2023, mostly gpt-3.5-turbo, which fails in ways newer models do not. These are not professional conversations in the benchturn sense. The study is English only, by the dataset's own label, and the domain split is a keyword heuristic. The sample gives the correction share a standard error of 0.9 percentage points and the either share 1.2.

What this means for training data

Even in casual, anonymous, unpaid chat, roughly one multi-turn conversation in twelve has the user telling the model it is wrong, and one in five has a correction or a redirection. In code, where the output can be executed, it is one in six. That is a meaningful rate for a corpus nobody built to collect corrections, and a floor rather than a ceiling. In professional work the corrections are richer: the accountant says which election and which year; the engineer attaches the failing test. The provenance WildChat cannot supply, who the user is and whether they know the field, comes free in a professional setting.

None of it currently reaches a training run with consent and payment attached. The logs sit with the vendor, stripped of the context that would make a correction verifiable, and the person who produced it is neither asked nor paid. This study puts a number on that signal in the least favourable setting we could measure. If you do work where you correct models every day, or you train models and want corrections you can trust, write to hello@benchturn.com.

Script and results: scripts/research/wildchat_corrections.py, content/research/data/wildchat-corrections.json