I publish a small Turkish narrative corpus that ships a machine-generated annotation layer — six binary craft features per scene. Those flags were never validated against human judgement, so I ran three studies to check them, and the result is unflattering enough that I want it on the record rather than buried.
Setup: 120 scenes scored against my own blind labels (Study 1); then 100 disjoint scenes scored against an independent volunteer’s labels, locked before any machine ran (Study 2: rule-based detector, Gemini 2.5 Flash, Grok; Study 2b: Claude Fable 5 High, ChatGPT 5.5, identical protocol).
The finding, on the one rule that requires real inference — whether an abstract state has been rendered as a concrete physical detail:
| Grok | Gemini 2.5 | ChatGPT 5.5 | Detector | Claude Fable 5 | Human | |
|---|---|---|---|---|---|---|
| Positives / 100 | 0 | 1 | 40 | 72 | 78 | 9 |
Cohen’s κ at or near chance for five of six labellers (0.004, 0.015, 0.000, 0.019, 0.027). Meanwhile ChatGPT posted the highest raw agreement in any study, 84.5% — because five of six rules have lopsided human distributions (positives: 0/1/9/96/99/44), so raw agreement mostly measures willingness to say “absent.” Four of my six rules were effectively not tested at all. That is a defect of my evaluation set, not of the models.
Two readings survive and I cannot separate them with one human rater per study: either the feature genuinely requires inference beyond current automatic raters, or the definition is not operational enough for anyone — including my human — to apply consistently. My own criterion drifted mid-pass in Study 1, which is evidence for the second.
What I would actually like from this forum is a second independent human rater. Everything needed is open: 100 scene texts, six definitions, locked human labels, all model label files, the ten prompt blocks, ID mapping and scoring scripts, in evaluation/.
Paper: How Reliable Are LLM Annotations? A Three-Study Benchmark Dataset: leventbulut/objective-projection · Datasets at Hugging Face
Disclosure: I wrote the rules and was the rater in Study 1. Claude is both one of the scored systems and was used to prepare the analysis scripts and the paper — the arithmetic is reproducible from published files, the framing is not neutral.