Behind each benchmark win is a room of people grading answers. The tasks have shifted from drawing boxes around cats to judging whether a legal summary is defensible, and the workforce has shifted with it: fewer click workers, more domain experts on hourly contracts.
From volume to judgment
Vendors say the average task now takes twelve minutes instead of twelve seconds. That changes pricing, tooling and who gets hired. Several firms have started recruiting retired accountants, nurses and paralegals to score model output in their old fields.
Key takeaways
- Expert graders earn three to five times the old labeling rate
- Quality audits now consume a third of project budgets
- Model labs are bringing part of the work in-house
This story is part of our ongoing coverage of ai. We will update it as more details emerge.
