Why a record, not a policy

We make a small number of promises: contributors choose what to share, we verify it, strip anything personal or proprietary, check it against public evaluations, and license it with recorded consent; contributors are paid when their work is licensed and can withdraw anything unshipped. A policy document can state all of that. It cannot show that it happened for any particular item.

A lab that licenses material from us should be able to answer, for any single item, where it came from, what the contributor agreed to, what was removed, and whether it overlaps a public benchmark, without contacting anyone. A contributor should be able to answer what was used and what it earned, also without contacting anyone. Both are queries against one record. This note describes that record as it exists in our first version. We expect to revise it.

The item

A contribution is a set of items. Each item is one unit of work: for example, a prompt, the model's first attempt, the contributor's correction with a stated reason, and the check they ran to confirm it. Everything else attaches at the item: consent, redaction, decontamination and payment are all recorded per item.

The item identifier is a hash of the content after redaction, so a lab holding the material can recompute it and confirm it matches the record. We also keep the hash before redaction, so we can attest that a redacted item derives from a given original without the original being in the licensed set.

{
  "item_id": "sha256:4f1c…",
  "contribution_id": "ctb_01j8x…",
  "contributor_id": "ctr_9a3e…",
  "domain": "contract-law",
  "created_at": "2026-07-22T14:03:11Z",
  "kind": "correction",
  "content_hash_before_redaction": "sha256:b72d…",
  "content_hash_after_redaction": "sha256:4f1c…",
  "redaction_log": {
    "names": 2,
    "identifiers": 1,
    "client_data": 3,
    "proprietary_code": 0
  },
  "verification": {
    "method": "professional-registration",
    "verified_at": "2026-06-30T09:15:00Z"
  },
  "consent": { "…": "see below" },
  "decontamination": { "…": "see below" },
  "licences": ["lic_0042"]
}

The contributor identifier is pseudonymous and stable across contributions, so a lab can see that two items came from the same person without learning who that is. The domain comes from a controlled vocabulary we maintain (tax, radiology, contract-law, software, and so on) rather than free text, so queries by domain are complete. The kind field records whether the item is a correction of a model output, a steering instruction, a hard case, a check, or a demonstration of the contributor doing the task themselves. The redaction log records categories and counts, never the removed content. The verification field records how we confirmed the contributor works in the field they claim, as a category rather than as evidence, and when.

Consent, per item and per purpose

Consent is not a single flag. A contributor may accept training but not public demonstration, or evaluation but not training. So consent is recorded per purpose, each carrying when it was granted and, if it has been, when it was revoked.

{
  "consent": {
    "training":      { "granted": true,  "granted_at": "2026-07-22T14:05:40Z", "revoked_at": null },
    "evaluation":    { "granted": true,  "granted_at": "2026-07-22T14:05:40Z", "revoked_at": null },
    "demonstration": { "granted": false, "granted_at": null, "revoked_at": null },
    "research":      { "granted": true,  "granted_at": "2026-07-22T14:05:40Z", "revoked_at": "2026-08-02T10:12:07Z" }
  }
}

The rule is that a purpose not present in the record is not granted. There is no default and no inheritance from the contribution or the contributor's account. If we add a purpose later, existing items do not acquire it.

Withdrawal sets revoked_at on the relevant purposes. An item already included in a signed licence stays in that licence: we have made a commitment to the licensee and do not unwind it. Everything else is excluded from future sets from the moment of revocation. The record shows both facts: the revocation timestamp on the consent entry, and the licence identifiers in the licences array, so a reader can see that the item was withdrawn and that it remains in one earlier licence. We tell contributors this before they share anything.

Decontamination

Every item is checked against a maintained list of public evaluation sets before it can enter a licensed set, using exact matching on normalised text and near-duplicate matching for paraphrases and partial overlaps. The record stores the list version, the check version, when it ran, and the result.

{
  "decontamination": {
    "list_version": "2026-08-01",
    "check_version": "0.3",
    "checked_at": "2026-08-03T02:41:19Z",
    "result": "clear",
    "matches": []
  }
}

This lets a lab say, for any item, which benchmarks it was checked against and when, rather than relying on a general assurance that checking happens. We can re-run items against a newer list and record the second check without losing the first.

The limits are real. Near-duplicate detection catches paraphrase up to a threshold and misses it beyond that; it is not a proof. The list has to be maintained by hand, and a benchmark published after an item was checked will not appear in its record until we re-run it. A clear result means the item did not match the list we had on the date shown, not that it matches nothing.

Licences and the ledger

A licence is a record of one set of items going to one licensee under one version of our terms.

{
  "licence_id": "lic_0042",
  "licensee": "lse_7f21…",
  "set_id": "set_2026q3_contract_law",
  "terms_version": "1.0",
  "signed_at": "2026-08-11T16:00:00Z"
}

The licensee is an opaque identifier. A contributor can see that their item is in a licence but not which lab holds it, and a lab's record does not name other licensees.

Payment is a ledger with one entry per item per licence.

{
  "item_id": "sha256:4f1c…",
  "licence_id": "lic_0042",
  "amount": "…",
  "currency": "EUR",
  "settled_at": "2026-08-29T00:00:00Z"
}

A contributor statement is a query over the ledger filtered by contributor: every item of theirs, every licence it appears in, what each pair earned, and when it settled. A lab audit is a query over licences filtered by licensee. In this version we keep per-item records but settle in pooled batches, so settled_at on a group of entries will usually match. The per-item granularity is what makes the statement answerable; pooled settlement is a practical choice we may change.

What a lab audit looks like

Did everything in a set carry training consent at the time of signing? Query the items in that set_id, read each consent record for the training purpose, and compare granted_at and revoked_at against the licence's signed_at.

Which items in a set have since been withdrawn? The same items, filtered to those with a non-null revoked_at on any purpose the licence covers. They remain in the licence, and the record shows why.

Does a particular item overlap a benchmark published last month? Read the item's decontamination record. If its list version predates the benchmark, the honest answer is that the item has not been checked against it, and the lab can ask us to re-run.

What was removed from an item before it arrived? The redaction log: categories and counts, plus the two content hashes showing that the item derives from an original we can attest to.

Open problems

We do not have settled answers to the following.

A contributor in a rare domain may be identifiable from the domain and the content of their work even after names and identifiers are removed. A pseudonym does not help if only a handful of people could have written the item. We are considering coarser domain labels and minimum group sizes for release, and do not yet know what that costs in usefulness.

Verifying that a contributor works in the field they claim requires evidence, and evidence is personal data. We want to hold as little as possible and record only the category of verification. Where the line sits between enough verification and too much collection is not resolved.

Near-duplicate detection across languages is weak. A benchmark item translated into another language will usually pass our current check.

When a licence is later disputed, by a contributor who says consent was recorded wrongly or a licensee who says an item should not have been included, the record shows what was recorded and when, but does not by itself decide who is right. We need a process for that and have not written it.

Write to us

If you are a lab or a contributor and you want to see the record for real, or you think one of the queries above does not actually answer the question it claims to, write to hello@benchturn.com. We are especially interested in hearing from people who want to poke holes in it.