We Published Our AI Rubric Calibration Data. Everyone Told Us Not To.
The AI judge scored 60% of responses at Level 4. Manual review put the real number at 15%. We published the gap.
The function returned a 200 response. The database was empty.
I was reviewing code that an AI assistant had generated for a notification processing system — an enterprise B2B platform that sends scheduled updates to thousands of customers. The code implemented the Unit of Work pattern, or appeared to. The variable names suggested it. The structure followed the pattern’s conventions. A reviewer scanning for architectural compliance would have approved it.
The AI had generated a database context, opened a transaction scope, performed the insert operations, and returned a success response. It had omitted the single line that commits the transaction — the CompleteAsync() call that flushes the staged operations to the database. Without it, the transaction scope disposes on exit, the staged operations are silently rolled back, and the database receives nothing.
The code was syntactically correct. It compiled. It ran without exceptions. The API returned 200 OK. The tests passed — because the tests used mocked database contexts that don’t enforce transaction semantics. Everything a reviewer would check confirmed that the code worked. The only signal that it didn’t was the empty database, and you had to look at the database to know that.
I caught it because I ran the endpoint and checked the actual data store. Not because the tooling caught it. Not because the tests caught it. Because I didn’t trust the 200.
Six months later, I encountered the mirror image of this problem in a different system.
I was building an autonomous agent platform — a production system where AI agents execute tasks, store results, and update a shared knowledge base. One of the agents had a bug in its output formatting — a function that assembled structured data from component parts. The function had a NameError: a variable referenced in the return statement didn’t match the variable assigned three lines above. A typo. The kind of bug a linter catches, the kind an experienced developer spots in seconds.
Thirty-one tests covered that agent. All thirty-one passed.
Polished Decoration
Every test used mock data and mock dependencies. The mocks replaced the actual function call with a pre-defined return value, which meant the function containing the NameError was never executed during testing. The tests verified that the agent’s orchestration logic worked — that it called the right functions in the right order with the right parameters. They never verified that the functions themselves produced correct output.
One hundred percent method coverage. Zero coverage of the runtime execution path that contained the bug.
The test suite wasn’t a safety net. It was polished decoration — an artifact that reported green and produced confidence without producing safety. The distinction matters because the confidence was the actual danger. If the tests had been absent, someone would have manually verified the function. The presence of a comprehensive-looking test suite eliminated the instinct to look deeper.
The test suite wasn't a safety net. It was polished decoration — an artifact that reported green and produced confidence without producing safety.
These two incidents — the committed transaction that wasn’t, and the test suite that covered everything except the bug — both trace to the same root cause. AI-generated code is confidently wrong in a way that human-written code usually isn’t.
When a junior developer writes bad code, the badness is usually visible. Variable names are inconsistent. Patterns are unfamiliar. The structure feels off. A reviewer’s pattern-matching instinct fires and they look closer. AI-generated code doesn’t trigger that instinct because it’s optimized for exactly the patterns reviewers look for. Clean naming. Consistent structure. Familiar architecture. The output is shaped to pass review, not to be correct.
A Windows installer project made this pattern structural for me. I was using AI assistance to build MSI packages with the WiX Toolset. The AI consistently generated XML that compiled without errors, produced valid installer packages, and installed cleanly on test machines. It also consistently used syntax from WiX versions 4 and 5 in a version 6 project.
The syntax compiled because WiX 6 maintains partial backward compatibility. It installed because the deprecated attributes still functioned at runtime. But the generated packages didn’t support the v6 features I needed — features the AI couldn’t use because it kept defaulting to the syntax it had seen most frequently in training data.
Same axis, repeated failures. Not random errors across different areas — systematic blind spots on the same version boundary, because the AI’s training data over-represented the older versions. The AI wasn’t occasionally wrong. It was reliably wrong about a specific thing, and reliably confident about it every time.
The content pipeline for a multi-brand platform added a dimension I hadn’t considered: AI overriding constraints in edge cases.
The system had a compliance gate — a set of rules governing what language could appear in customer-facing content. The AI-generated copy passed the compliance check on every main body section. Clean, compliant, well-within-guidelines.
The violations appeared in callout boxes. In sidebar text. In caption copy for graphics. In the decorative elements that surround the primary content — the places where a compliance reviewer’s attention is lowest and the AI’s pattern-matching is most likely to drift.
Where Violations Hide
The main body was compliant because the compliance prompt was optimized for main body content. The peripheral elements were generated with lighter oversight because they were “just formatting.” The AI didn’t violate the rules deliberately. It produced compliant text where the constraints were explicit and drifted where the constraints were implicit.
I now maintain what I call an Override Log. When the AI is confidently wrong — not randomly wrong, but wrong in a pattern — I document what it produced, why it looked right, how I caught it, and what I changed. The Unit of Work omission. The mock-heavy tests. The WiX version drift. The compliance edge-case violations. Each entry is a named failure pattern, not an anecdote.
Over time, the log transforms from a reactive record into a predictive instrument. You stop catching these failures after the fact and start anticipating them structurally. The question shifts from “is this code correct?” to “what class of correctness can this AI reliably produce, and where does its reliability drop?”
This is the principle that drove the decision to publish the Comprehension Audit calibration data.
The LLM judge that powers the Comprehension Audit had the same confidence problem. Before calibration, it scored 60% of test responses at Level 4 or higher. Manual assessment of those same responses — by someone who has built the systems these questions describe — put the real number at 15%.
The judge wasn’t broken. It was doing what every language model does when asked to evaluate: drifting toward the agreeable middle, producing scores that feel right without being precise. “This response demonstrates some understanding of failure modes” gets a 4. “This response mentions failure modes in passing without demonstrating operational comprehension” should get a 2. The uncalibrated judge couldn’t tell the difference.
I built 24 calibration examples — three per dimension, at score levels 2, 3, and 4. Not the extremes. Not the obvious 1s and the obvious 5s. The boundary cases where the judge drifted most. Each example includes not just the score but a “why this score” commentary that explains the specific textual signals that distinguish a 2 from a 3, a 3 from a 4. The commentary is the calibration — it teaches the judge what to look for, not just what number to assign.
Then I published all of it. Open source. The rubric, the calibration examples, the scoring methodology, the weight distribution.
The counterargument was obvious: this is competitive moat. Anyone could take the calibration data, build a competing tool, and skip the months of iteration that produced these examples. Why give it away?
The answer is the same principle that drove the Override Log: transparency produces better outcomes than secrecy when the goal is industry-wide improvement. I’m not building a diagnostic tool to protect a competitive advantage. I’m building it because the industry needs a shared standard for measuring AI project comprehension — and a standard that lives behind a paywall isn’t a standard. It’s a product.
A standard that lives behind a paywall isn't a standard. It's a product.
The calibration data is the override made permanent. It takes the pattern — AI is confidently wrong, human catches it, human documents the correction — and publishes the correction as a public resource. Anyone building an AI evaluation system can use these examples to calibrate their own judges. That’s not a competitive threat. That’s the point.
AI Override Discipline isn’t about distrusting AI. It’s about understanding the specific ways AI confidence diverges from AI accuracy — and building systems that catch the divergence before it compounds.
The committed transaction that wasn’t. The test suite that covered everything except the bug. The version syntax that compiled and failed silently. The compliance copy that was clean in the body and violated in the margins. The evaluation judge that scored 60% at Level 4 when the real number was 15%.
Every one of these failures had the same signature: the AI was confident, the output looked correct, and the error was invisible to the review process that was supposed to catch it. The override discipline isn’t “check the AI’s work.” It’s “know where the AI’s work is most likely to be wrong, and build verification systems that target those specific failure zones.”
Trust the AI’s competence. Verify its confidence. Publish the gap.
The Comprehension Audit applies AI Override Discipline to evaluation itself — calibrating an LLM judge against documented failure patterns to produce scores you can trust. Explore the full calibration dataset in the open-source repository.
Wilfred Morgan
AI Systems Architect · Agentic AI Implementation