Post-training is where models are won

01What post-training data actually does

Post-training covers several kinds of data, each teaching the model something different:

• Supervised fine-tuning (SFT). Prompts paired with expert-written responses. This teaches the model what a good answer looks like in a domain, including format, depth and tone.
• Preference data for RLHF and DPO. Pairs of responses where a reviewer marks which is better and why. This teaches the model to distinguish between answers that are both plausible.
• Rubric-based evaluations. Held-out items scored against explicit criteria, used to measure progress and catch regressions.
• Red-team transcripts. Adversarial prompts and graded responses that show where the model is unsafe or easily misled.

Of these, preference data has the highest leverage and the highest risk. Every ranked pair is a gradient signal. If the rankings are inconsistent or wrong, the model is optimised toward the reviewers' confusion.

SFT usually comes first: it shows the model what a good answer looks like. Preference data then teaches it to choose between answers that all look acceptable. RLHF and DPO use the same ranked pairs in different ways. RLHF trains a separate reward model on them and then optimises the model against it with reinforcement learning. DPO skips the reward model and trains on the pairs directly, which is simpler and more stable to run. Where correctness can be checked by a program, such as tests passing or a verified final answer, teams increasingly add reinforcement learning against those checks as well, which is what environments with graders are for.

02Why crowd ranking fails at the frontier

Early RLHF work could use generalist reviewers because the comparisons were easy: one response was coherent and the other was not. Frontier models rarely fail that way any more. Their mistakes are subtle: a drug interaction that is mentioned but understated, a refactor that compiles but introduces a race condition, a legal summary that is accurate except for the one clause that matters.

Spotting those mistakes takes domain expertise. A generalist reviewer looking at two fluent answers will fall back on surface signals like length, confidence and formatting. Optimise against those preferences and you get a model that is verbose, overconfident and wrong in exactly the places it matters most.

How much preference data you need depends on the domain and how far the model is from where you want it. Published recipes range from tens of thousands of comparisons (one well-known reward model was trained on about 33,000 prompts) to more than a million for a large open model. In a specialist domain, a pilot batch of a few thousand expert pairs with measured agreement tells you more than a large crowd batch, because noisy pairs add volume without adding signal.

A reviewer who cannot tell which answer is correct will choose the one that sounds correct. The model learns the difference.

03The rubric is the product

We staff post-training projects with domain experts: engineers for code, clinicians for medicine, linguists for language work. More importantly, we train them against the client's specific rubric, not a generalised labelling guideline.

A rubric is where a lab's view of what good looks like gets written down. It is also the most common source of quality problems, because rubrics are ambiguous in ways nobody notices until two reviewers disagree. The process we run treats the rubric as a living, versioned artefact:

• Calibration rounds before production work, where reviewers label the same items and disagreements are resolved by clarifying the rubric, not by majority vote.
• Two independent reviewers per preference pair, so agreement can be measured instead of assumed.
• Agreement thresholds per batch. Batches below the agreed threshold are redone, not shipped with a caveat.
• Versioned rubrics. Every label records the rubric version it was made under, and the rubric ships with the data.

Every batch ships with reviewer-agreement metrics, so quality is measurable, not asserted.

04Keep the reason

A preference pair that only records which response won throws away most of the information the reviewer produced. We keep the reason for every rejection. In a code pair, that might be that the rejected answer wraps a loop in a try/except and hides the failure instead of fixing it.

Reasons serve three purposes. They make reviewer disagreement diagnosable, because two reviewers who disagree for different reasons point to a rubric gap. They support training methods that use critiques as well as preferences. And they let a lab audit a batch long after it was made.

05Drift is the silent failure

Post-training projects run for months. Over that time reviewers change, rubrics get amended and the model being evaluated gets better, which makes comparisons harder. Without monitoring, quality drifts and nobody notices until a release feels worse than the last.

We track agreement and gold-standard accuracy over time, re-run calibration when the rubric changes, and keep the same review standards across sourcing, ranking and evaluation so guidelines never drift between vendors. Teams that treat human data as an engineering discipline, with versioned rubrics, calibration rounds and drift monitoring, are the ones whose models feel meaningfully better with each release.

06Evaluation closes the loop

The last piece is measuring whether any of it worked. Public benchmarks are useful for comparison but increasingly contaminated and rarely aligned with a specific product. We build expert-written evaluation sets with a held-out answer key, checked against public data for contamination before release and scored by domain specialists against a rubric.

The same experts who write your preference data can write the evals that test it, under the same rubric. That consistency is what turns post-training from a series of experiments into a process you can improve. See our conversational vertical for the formats we deliver.