The Failure Mode Taxonomy: 13 Ways Frontier Models Reason Badly, and How to Catch Each One

In the research writeup I said the failure mode taxonomy was one of the few things from the program I’d be comfortable publishing in full, and that I probably would, soon, as its own post. This is that post.

The single most useful artifact from a year of watching frontier models reason is a catalog of the specific, repeatable ways they reason badly. Not “limitations” in the marketing sense. Mechanism-level patterns that show up again and again once you watch long enough: the same wrong moves, across Claude, GPT, Gemini, and Grok, across problem types, across model generations. Thirteen of them eventually got stable names.

The reason to bother naming them is that the most damaging failures are the ones that look fine. A wrong answer with no structure gets caught. A wrong answer wrapped in clean headers, confident phrasing, and a citation nobody checks usually does not, whether the reviewer is a human skimming or an automated grader scoring only the final answer. Most of this catalog is about the failures that pass inspection.

One thing up front, because it shapes how to read the rest. Naming these modes is not novel. Several are widely recognized by now, and anyone who has spent real time with these models has seen something like hero bias or formatting theater, named or not. The catalog itself is not the differentiator. What compounds is that every mode is paired with a countermeasure, and the countermeasure fires automatically, on every architecture, every run. The value was never in noticing that models hallucinate compliance. It is in having a patch for it that runs whether or not anyone is watching.

So this post does two things for each mode: it describes the failure plainly enough that you can spot it in your own outputs, and it describes the class of countermeasure that works against it. I am deliberately describing countermeasures at the level of principle, not implementation.

Two classes: score-affecting and metadata-only

The thirteen modes split into two groups, and the split matters more than it looks.

Score-affecting modes (5) are real quality failures. When one fires, the output is worse, and an evaluation should dock points on the relevant dimension (correctness, reasoning, completeness). These are the ones that change whether an answer is right.

Metadata-only modes (8) are behavioral patterns worth tracking but not automatically disqualifying. A response can exhibit one and still be correct. They are diagnostic signal about how a model is reasoning, useful for spotting weak spots and for predicting where a model will break, but you don’t deduct for them directly. You log them.

The reason to keep these separate is that conflating them corrupts your scoring. Penalize formatting theater as if it were a wrong answer and you punish well-organized correct work. Ignore hallucinated compliance because “the response said it did the thing” and you reward confident fabrication. The whole point of a taxonomy is to score the substance and merely note the style.

The score-affecting five

These five indicate substantive failure. When detected, they cost points.

Core Challenge Missed

The failure. The model applies a generic template to a unique problem. The answer would work just as well for a different question with similar surface features. The specific thing that makes this problem hard, the constraint that breaks the standard approach, never gets addressed. The output is well-structured, fluent, and wrong, because it answered the category of problem rather than the problem.

This is the most common failure mode across every architecture I have tested. Models with heavy textbook training are slightly more prone to it, because they have more templates to reach for.

How to spot it. Ask one question: would this response work just as well for a different problem with the same keywords? If yes, the model pattern-matched the category and skipped the specifics. Generic frameworks applied to unique constraint sets are the signature.

Countermeasure. Force the specific constraint to the surface before any solving happens. The technique that works is requiring the solver to name what makes this particular problem hard, the condition that invalidates the textbook move, before it is allowed to commit to an approach. A problem whose unique constraint has been stated out loud is much harder to answer generically. Pairing this with a separate routing or classification step, one whose only job is to identify the trap rather than solve, catches a large share of these before they start.

Hallucinated Compliance

The failure. The response claims to satisfy a constraint that it demonstrably does not satisfy. It asserts compliance instead of demonstrating it. This is the most dangerous mode in the catalog, because it passes casual inspection: the assertion of compliance is exactly the thing a hurried reader checks for, and it is right there.

Examples that recur: a citation to “lines 251 to 370” of a file that is 70 lines long; an answer that says “as required, in the specified format” while quietly violating the format; a claim self-labeled as the highest evidence tier whose own parenthetical admits the evidence is hearsay. The self-certifying checkmark, the ”✅ verified” attached to a step nobody verified, is the same mechanism wearing a different costume: it is a compliance claim about the act of verification itself.

How to spot it. For every explicit constraint in the task, run a binary check: did the response actually satisfy it, or did it only claim to? The claim is not the evidence. Pay special attention to citations, numeric requirements, and format conditions, since those are checkable in seconds and fail silently.

Countermeasure. Treat every claim of compliance as unverified until checked against the source. The countermeasure class is a verification pass that re-derives compliance independently rather than trusting the response’s own account of itself, and that checks cited references against what the references actually say. The governing principle, which generalizes well beyond this one mode, is demonstrate, do not assert. An output earns the “I satisfied X” sentence only by showing the satisfied X.

Self-Contradictory

The failure. The response contradicts itself within the same output, without noticing. It asserts one thing in the opening and the opposite in the body, or states that a system is consistent and then spends three paragraphs surfacing its inconsistencies, then returns to “consistent” in the conclusion as if nothing happened. Longer responses are more prone to it, because there is more surface area for the contradiction to hide in.

How to spot it. Scan for assertions that are negated elsewhere in the same text. Are the numbers consistent throughout? Does the summary agree with the body it summarizes? Do the early conclusions survive the later analysis? This is partly automatable: look for negated variants of the response’s own key claims.

Countermeasure. Self-review does not catch this reliably, because the same context that produced the contradiction tends to smooth over it on a reread. The countermeasure that works is an independent pass with fresh context whose explicit job is internal consistency, plus a mechanical reconciliation check on any numbers that appear more than once. Separation of context is the load-bearing part. A reviewer that did not write the answer has no investment in its framing and reads the contradiction at face value.

Hero Bias

The failure. The response overweights evidence for the conclusion it “wants” and bends mixed or negative evidence toward a positive outcome. In a narrative setting it looks like assuming the protagonist wins because that is how stories usually end. In a research setting it looks like the analysis drifting toward whatever confirms the thesis the system is already invested in. The conclusion the model reaches is the one it preferred, not the one the evidence supports.

This one is especially dangerous when the evaluating system has a thesis of its own. I call that pull thesis gravity: the tendency of a system studying its own research program to keep concluding that the program works. My own pipeline once claimed “three independent designs converged on identical outcomes” when only two of the three actually did. Two-of-three is still interesting. It is not three-of-three, and the narrative wanted three.

How to spot it. Apply the weakest-link test: would the conclusion change if you removed the single weakest piece of supporting evidence? If yes, it was resting on insufficient support. And check whether counterevidence is engaged with or quietly minimized.

Countermeasure. The countermeasure is an adversarial pass that is specifically tasked to find the conclusion the evidence does not support, run with enough independence that it is not protecting the original answer’s framing. General confirmation-bias review is too weak; the reviewer has to be pointed at thesis gravity by name and rewarded for puncturing the preferred narrative rather than for agreeing with it.

Discount Logic

The failure. The response dismisses an edge case, boundary condition, or constraint as “minor,” “unlikely,” “negligible,” or “physically unrealizable,” with no justification, and the reasoning that follows is sound for the general case but collapses exactly where it was waved off. Often the dismissed edge case is the core challenge. The model correctly identifies the boundary condition, labels it irrelevant, and builds an otherwise clean analysis on the incorrect dismissal.

Models with strong scientific training are more prone to this, having absorbed the habit of throwing out “unphysical” scenarios, sometimes the very scenario the problem was built around.

How to spot it. Every time the response dismisses something, check whether it was actually addressed or just hand-waved. Then probe the reasoning at the boundaries: does the logic still hold when a variable is zero, negative, infinite, or at its stated limit? Does the response say where it breaks down, or assert robustness it never demonstrates?

Countermeasure. Any dismissal has to be justified against the problem’s stated constraints, not against a general prior about what is “usually” negligible. The countermeasure class is explicit boundary probing, testing the reasoning at the limits rather than the middle, and refusing to accept “this case doesn’t matter” unless the case has been shown not to matter under the actual conditions given.

The metadata-only eight

These eight are diagnostic. They tell you how a model is reasoning and where it is likely to break, but a response can exhibit one and still be correct. Track them; don’t auto-penalize them.

Formatting Theater

The failure. Impressive structure, thin substance. Clean headers, tidy bullets, a summary table, and underneath it two or three sentences of surface observation per section. It reads like an outline that was never filled in. It is dangerous in a specific way: length-biased and structure-biased graders, human and model alike, reward it, so it inflates confidence in the evaluator, not just the response.

It is the most common metadata mode, and its most common partner is Core Challenge Missed. Structure is what a response reaches for when it has nothing to say about the actual problem.

How to spot it. Compare presentation quality to correctness. If the structure score is far above the content score, flag it. Ask whether the headers promise depth the body does not deliver.

Countermeasure. Decouple structure from substance in scoring so polish cannot stand in for correctness. The simplest version, useful even for manual review, is to strip the formatting and reread the plain content: what survives without the scaffolding is the actual answer.

Mystical Obfuscation

The failure. Dense, sophisticated-sounding language that hides a lack of substance. Terms get introduced and never tied to a concrete referent. It sounds deep without saying anything specific. More common on philosophical or theoretical questions, and more common in models with broad academic-text training, which have seen the pattern and can reproduce its texture fluently.

How to spot it. For each conceptual term, ask whether it is defined and whether it is used with a concrete referent. Could you explain what it means based only on how the response uses it? Terms introduced and then abandoned without application are the signature.

Countermeasure. Require a concrete referent for every term that carries weight, and run an “explain it again without the jargon” pass. Anything that cannot survive being restated in plain language was decorative.

Jargon Inflation

The failure. Invented terminology that sounds precise but is never grounded. The response coins a name that inflates a simple point into something that sounds like a formal concept. From my own pre-codification era: a straightforward observation about governed agents refusing harmful requests once got named “Self-Canceling Threat Vector Equilibrium.” The observation was fine. The name was a physics costume on a plain logical point.

The distinction from Mystical Obfuscation: obfuscation uses language to hide; inflation uses invented terms to impress. They can co-occur.

How to spot it. For each coined or unusual term, check whether it comes with a concrete example. Could someone unfamiliar understand the referent from context? A term introduced, used once or twice, and never grounded is the tell.

Countermeasure. A coined term has to earn its name with a grounded example, and naming something is not allowed to substitute for demonstrating it. In a research context this is worth watching closely, because impressive terminology is often quietly in the service of hero bias: a finding with a grand name feels more significant than it is.

Kitchen Sink

The failure. Everything thrown at the problem with no prioritization. The response addresses ten-plus sub-points at equal weight and no triage, producing breadth with no depth anywhere and no signal about which points actually matter. More common on broad open-ended questions, and more common in long-context models that have room to be exhaustive instead of selective.

How to spot it. Count the distinct claims. If there are ten or more at roughly equal depth with no explicit ranking, and the response never says which points are load-bearing, that is the pattern.

Countermeasure. Force triage. Require the response to rank or select rather than enumerate, and to state which points are most important and why. A cap on how much can be included without prioritization turns inclusion-by-default back into selection.

Confident Wrong

The failure. An incorrect answer asserted with no uncertainty flagging. Not just wrong, but wrong and unhedged, with no caveat or acknowledgment that the claim could be off. The signature is the absence of uncertainty language on a claim that turns out false. “X is true” when X is false is Confident Wrong; “X appears true based on Y” is not, even if the conclusion is incorrect, because the uncertainty was acknowledged.

How to spot it. When a response makes a strong, unqualified assertion, verify it, especially anything numeric or factual. The combination of high confidence and an easily-checkable false claim is the thing to catch.

Countermeasure. Verify strong unqualified assertions rather than letting confident tone carry them, and compute-verify anything numeric instead of trusting stated arithmetic. The deeper fix is calibration discipline: uncertainty language should scale with how thin the evidence actually is, so that the confidence in the prose tracks the confidence the evidence supports.

Paradox Inflation

The failure. Faced with a self-referential problem, the response defaults to declaring it a paradox or impossible, even when the system is consistent and resolvable. It pattern-matches on “self-referential” and reaches for “paradox” instead of doing the logical work of finding the fixed point. Self-reference questions are among the most discriminating in my benchmark set precisely because architectures default to claiming paradox at very different rates.

How to spot it. When the response claims a paradox, check whether the system is actually contradictory or just consistently resolvable. Many self-referential systems have stable fixed points that are not paradoxes. A genuine paradox claim should demonstrate the contradiction, not assert it.

Countermeasure. A paradox claim has to be earned by exhibiting the actual contradiction, and the resolution has to be searched for first. The countermeasure class forces the resolution attempt before the paradox label is allowed, which turns “this is impossible” from a default escape hatch into a conclusion that has to survive an honest search for the fixed point.

Echo Without Analysis

The failure. The response restates the question and the provided information without adding reasoning. Strip the paraphrase and very little original analysis remains. More common on shorter responses and on questions that arrive with a lot of their own context, since the model can use the provided material as “safe” content that displaces actual thinking.

How to spot it. Mentally remove every sentence that paraphrases the question or restates given information. What is left? If the remainder is a small fraction of the response and contains no original analysis, that is the mode.

Countermeasure. Measure original contribution by subtracting the echo. The useful discipline is to require that the analysis build on the input rather than summarize it, and to treat restatement as setup that does not count toward the answer.

Format Violation

The failure. The response ignores explicit format requirements. Asked for JSON, it returns prose. Asked for exactly five items, it lists seven. Asked for a structured record with required fields, it merges two of them and drops a third. Unlike the rest of the catalog, this one needs no judgment to detect: a format constraint is either met or it is not.

Susceptibility rises with format complexity. Simple requirements are rarely violated; multi-field formats with conditional logic are violated often.

How to spot it. Check the output against every stated format constraint, one binary check per constraint. It is the most mechanical detection in the taxonomy.

Countermeasure. A mechanical per-constraint format check, applied before the output is accepted. Because the check is binary and judgment-free, it is the easiest mode in the catalog to automate completely, and it pairs naturally with the Hallucinated Compliance check, since a format violation is frequently accompanied by a claim that the format was followed.

How they cluster

The modes do not fire independently. A handful of co-occurrences show up often enough to be worth watching as units:

The practical upshot: catching one member of a pair is a reason to go looking for the other.

Why a single grader is not enough

One more pattern, because it changes how you should run detection at all. When different model architectures grade the same output, they do not just disagree on scores. They are systematically strong at catching different modes. One architecture reads for methodology and reasoning process and is sharp on self-contradiction and hero bias. Another reads for answer correctness against a canonical solution and is sharp on missed challenges and confident-wrong arithmetic. Others lean toward logical-proof verification or toward fairness and bias.

This divergence is signal, not noise. A pipeline that routes each architecture to the modes it is best at detecting gets broader coverage than any single grader can. It is also the cheapest credibility fix available to anyone doing comparative evaluation: rotating graders across architectures, rather than trusting one, removes a large source of single-grader distortion at almost no extra cost. (The grader-bias numbers behind this, including how much each architecture inflates its own family, are in the research post.)

A note on scope

This is a taxonomy from one corpus, not a law of model behavior. It came out of my own evaluation and agent-building work, so it carries the fingerprints of the problems I happened to test and the way I happened to grade them. The prevalence of each mode shifts with task type, model, scaffold, and judge. Some of the categories overlap, and with more data a few may merge into a single mechanism or split into subtypes. The thirteen names are stable in my own registry, but the catalog is open-ended: new modes get added as new model generations surface them.

These thirteen are also specifically failures of an output being graded. There is a related family I keep separate: failures of an agent at runtime. An arbiter rubber-stamping the most fluent of several disagreeing answers instead of resolving the disagreement. A long reasoning run decaying into confident repetition after the useful thinking has ended. A verification step quietly sliding into rationalization, explaining why a claim is plausible instead of actually checking it. Those are real, and the agent handles them, but they belong to a different taxonomy than the one here. I may write that one up separately.

The point is not that these are the final names. It is that reliability work needs this kind of vocabulary at all. Before I named Core Challenge Missed, it read as a vague sense that an answer was off; after, it was a check I could run on every output. That is the whole move: observe the failure, name it, build a test for it, wire in a countermeasure, then measure whether the countermeasure actually holds. If you cannot name the way a system fails, you cannot test for it.

About the post

Read this as a field guide rather than a blueprint. The detection ideas are usable as a manual checklist or as the starting points for automated checks. The countermeasure classes are real and generalizable: binary per-constraint verification, structure-content decoupling, adversarial passes with independent context, evidence tagging, boundary probing, forced triage, demonstrate-don’t-assert. What I have held back is the implementation: the exact triggers, thresholds, prompts, and routing that make these fire reliably inside the agent. That gap is deliberate, and it is the honest line between what is useful to share and what is the actual work.

If you are building evaluation infrastructure and any of this overlaps with what you are seeing in your own outputs, I would be glad to compare notes. The question I find most useful to put to other builders: where do you see models failing in ways that final-answer scoring does not capture?

Eugene Dvorochkin, Fieldframe Labs, independent AI behavioral research and infrastructure since May 2025. Contact: edvorochkin@gmail.com. The broader research program is here; the HLE writeup is here.