Crowd labeling optimises for throughput on tasks anyone can do. For years that was most annotation: drawing boxes around cars, transcribing clear speech, marking whether a sentence is positive or negative. Those tasks still exist, but they are no longer what moves frontier models.
The annotation that matters today, such as medical image segmentation, code review judgments and legal document classification, is not work anyone can do. And label noise from unqualified annotators does not average out. It teaches the model the annotators' confusion.
01Label noise does not average out
The intuition behind crowd labeling is statistical: get enough people to label each item, take the majority, and individual errors cancel. That holds when errors are random. It fails when errors are systematic, and errors from people who lack the domain knowledge are almost always systematic.
If most non-specialists misread a particular type of skin lesion, a majority vote does not fix the label. It locks in the wrong answer with high agreement, which looks like quality. The model then learns the misreading with confidence, and the error only surfaces when a specialist reviews the model's output, if anyone does.
High agreement among people who do not know the answer is not quality. It is consistent error.
02Where expertise changes the label
In specialist domains, the correct label often depends on knowledge a generalist does not have and cannot pick up from a guideline:
- Medical imaging. Where a lesion boundary sits, whether a shadow is an artefact, which finding is clinically significant.
- Code. Whether a change introduces a race condition, a security issue or a performance regression that the tests do not cover.
- Legal documents. Which clause changes the risk profile of a contract, and how a redline alters an obligation.
- Speech. Whether a word is a regional pronunciation or a mistranscription, in a language the reviewer speaks natively.
In each case, a generalist can follow the instructions and still get the label wrong, because the instructions cannot encode years of training.
03How we staff
We hire and train reviewers against the specific task. A dermatology labeling project gets reviewers trained in dermatology; a codebase evaluation project gets working engineers; a speech project gets native speakers of the language and, where it matters, the dialect.
Domain expertise is the entry requirement, not the whole qualification. Reviewers are then trained on the client's rubric and must pass calibration before they touch production work. A senior engineer who disagrees with the client's definition of a good code review needs to label to the client's definition, and calibration is where that gets settled.
04Quality you can measure
Quality is enforced with layered checks on every batch:
- Calibration rounds before production and whenever the rubric changes.
- Gold-standard checks, with items of known answers mixed into live work so accuracy is measured continuously, not just at onboarding.
- Inter-annotator agreement tracked per batch, with batches below the agreed threshold redone.
- Expert review by a domain specialist on every batch.
- Rejection, not patching. Records that fail checks are rejected with the reason attached, not quietly fixed and shipped.
Each batch ships with its agreement metrics and the rubric it was labelled under, so the client can verify quality instead of taking it on trust.
Agreement is usually reported as Cohen's or Fleiss' kappa, which corrects for agreement that would happen by chance. On the common Landis and Koch scale, 0.61 to 0.80 is substantial and above 0.80 almost perfect, but the right threshold depends on the task. A subjective preference rubric will sit lower than a lesion boundary, so the threshold should be agreed before production starts.
05The cost argument, honestly
Specialists cost more per hour than generalists. That is the whole case for crowd labeling, and it is weaker than it looks.
Specialists label complex items faster and more accurately, which narrows the per-label cost gap. Fewer labels need rework. Fewer redundant labels are needed per item, because agreement is high for the right reasons. And the expensive part of a bad label is not the label itself, it is the training run it goes into, the evaluation that misleads the team and the deployment that fails. The cost difference between a generalist labeler and a trained specialist is small. The quality difference compounds through every epoch of training.
Crowd labeling still has its place. When most people do a task correctly, such as drawing a box around a car or transcribing clear speech in their own language, a non-specialist's label matches an expert's and paying for expertise buys nothing. The case for specialists starts where a guideline cannot carry the knowledge the label depends on.
06Vendor or partner
A data vendor sells task throughput. A data partner owns the outcome. Our clients own outcomes, so we own quality end to end: sourcing, review, preference ranking and evaluation under one rubric, so guidelines never drift between vendors and nobody has to reconcile three different definitions of correct.
If you are planning an annotation project in a specialist domain, the fastest way to see the difference is a sample batch scored against your own gold set. Tell us the task and we will scope one.
