The research writeup ended with an open question. HLE rewards broad knowledge as well as reasoning, so how much of the governance lift was about reasoning structure, and how much was the model being walked back to things it already knew? There is one clean way to find out: take the same methodology to a domain where stored knowledge cannot help.
That is what the program has been doing since the spring. The domain is ARC-AGI-3, a benchmark of interactive visual reasoning games explicitly designed to resist memorization. Nothing in a model’s training data tells it how these games work. Every game has hidden mechanics that have to be discovered by acting in the world, and performance is graded against human baselines. It is as close to a pure test of reasoning structure as any public benchmark gets.
This post is not the results post. Results are coming once the evaluation cycle completes, and the plan is to submit for verification on the hidden set. This post is about the two things the build produced that I think are more interesting than any score: an agent architecture built around compiling a specific human’s reasoning into the system, and the verification machinery that grew because I kept catching the agent, and myself, being wrong in ways that polite inspection would have missed.
The thesis: expert reasoning compilation
The honest starting point is that the original FF-STACK was incomplete, and ARC exposed it.
The stack that produced the HLE result encoded a year of observed LLM failure modes with countermeasures: the thirteen-mode taxonomy, the evidence discipline, the verification passes. On a knowledge-heavy benchmark that was enough, because the missing ingredient, knowing things, was supplied by the model. On ARC it was not enough. Knowledge-free games do not need a model that avoids hallucinated compliance. They need a model that knows what to try, how to read an unfamiliar mechanic, when a stuck position means “wrong hypothesis” versus “right hypothesis, wrong execution.” That is not LLM-failure-mode knowledge. That is human solving cognition, and the original stack contained none of it.
So the ARC agent is built around a bet I have not seen any lab pursue: take one specific human who is verifiably good at these games, whose reasoning happens to be unusually legible, and compile that person’s solving cognition into the agent’s operating layer. Concretely, the human (me, in this instance) plays the public games while narrating decisions out loud: what I noticed first, what I tested, what rule I inferred, what order I solved things in and why. Those narrations get mined into general, provenance-tagged patterns: mechanics grammars (“an isolated object that belongs to no target set is often a transformer; drive a piece through it and diff what changed about the piece”) and planning grammars (“when every direct route self-blocks, stop routing the goal piece and start clearing blockers, in order”). The patterns are deliberately general. The boundary rule, which matters for competition eligibility and gets enforced by machinery described below, is that the agent receives grammars, never answer keys: how this class of mechanic tends to work, never the click sequence that beats level four.
Why bother, when the field’s default answer is “scale the model”? Because ARC-AGI-3 is graded against human baselines, and that grading choice quietly defines the moat. If the benchmark measures “can an agent do what a competent human does in an unfamiliar interactive world,” then codified human competence is not a hack. It is the substance of the test. The generalization question, whether grammars compiled from 25 public games transfer to games nobody has seen, is the experiment. The early causal evidence is what convinced me the bet is real: a wall that had failed repeatedly fell on the first fair test after one human insight was encoded as a general pattern, and the insight visibly fired in the reasoning trace at the exact decision point where the prior runs had failed. The full accounting belongs to the results post.
There is also a mirror worth naming. A growing product category is trying to build digital versions of people from their data: memory layers, personal knowledge bases, agents that act on your behalf. This program is running the inverse experiment, compiling a person’s reasoning rather than their facts, and it hits the same failure classes from the other direction. More on that below, because they are not hypothetical.
The architecture: a mind and a body
The agent splits along a line that the whole build kept re-teaching: language models are good at judgment and bad at bookkeeping, and deterministic code is the opposite.
The body is deterministic Python. It holds a faithful clone of the live game engine for consequence-free simulation, verified against the real engine and treated with documented suspicion anyway (more below). It does search: breadth-first and heuristic exploration over moves and click targets, with frame deduplication and subgoal sequencing. It does perception: segmenting the pixel board into objects with colors, shapes, and markers, rendering text grids, reading sub-cell detail on demand, tracking cameras and scroll. And it attempts a cold solve before the mind ever spends a token, because a meaningful fraction of early levels fall to pure search.
The mind is a frontier LLM in a native tool-use loop. It does not micromanage moves. It reads the board, forms hypotheses, and delegates: run this action sequence on a clone and tell me what changed; search toward this board pattern; execute this plan but verify it on a clone first and refuse it if it does not progress; here is my hypothesis for the mechanic, expressed as code, falsify it against the engine and tell me exactly which cells I mispredict. That last tool matters more than it looks: the mind’s mental model becomes a testable artifact instead of a vibe.
The prompt layer carries the compiled cognition: a short procedural spine (hypothesis-first, probe before acting, never conclude impossible, an impossibility proof indicts your model of the game, not the game), and the prior registry of mined grammars, delivered not as a monolithic blob but as a compact index plus the specific playbook relevant to the current game and level, each row carrying its provenance and evidence date.
That spine is not decoration; the agent obeys it in its own voice. On a recent run it ended early with live budget to spare, writing, verbatim: “This is a reasoned pause at a verified impasse, not a claim of impossibility — the solution requires surfacing a mechanic the exposed controls didn’t reveal to me this session.” It refused to burn irreversible moves on hypotheses its own clone experiments had already rejected. That is the compiled epistemics speaking, unprompted, and it is the only kind of evidence for a prompt layer that I trust: the principle showing up in the agent’s own words at a moment nobody engineered.
Two design laws govern the whole thing, and they were cross-validated on two unrelated programs (HLE first, ARC independently): trust-based callable tools help; prescriptive mid-solve nudges hurt. Every mechanism that tells the model what to do mid-reasoning (“conserve moves,” “you should reconsider,” blanket caps) tested net-negative on both programs. Every mechanism that gives the model a trustworthy instrument it can choose to invoke tested positive. The agent is built almost entirely out of the second kind, plus a small set of conditional one-shot interventions that each earned their place through a documented, receipted failure.
The memory system rounds it out: a persistent scratchpad the mind writes to itself across levels and restores, cross-run memory, and auto-compiled win traces. And it is exactly here, at the seam where human narration and machine memory meet, that the most instructive failure of the project so far occurred.
The failure that taught the mining rule
One game’s row in the prior registry described a claw mechanic: grab the platform, place it, cross, regrab, re-place. The word “place” came from my own narration. I meant “drive the claw, which is holding the platform, to where you need a bridge.” The mining process turned it into a release input. A grab/release toggle. A button that does not exist in the game.
The agent then did something that made the failure genuinely interesting instead of just embarrassing. Mid-run, it tested the claw exhaustively and proved there was no release input, wrote the correct finding into its own notes, and then kept “there must be a release input I haven’t found” as its top hypothesis anyway, because the prior said so. A real source, a faithful citation, a wrong fact, and the wrong fact outranked the agent’s own correct measurement. The run burned its budget hunting a phantom control while the intended route sat unexplored.
Two defenses came out of the postmortem, and they generalize well past this project:
On the mining side: transcribe moves, not metaphors. A mechanic claim is not allowed into the registry on the strength of narration alone; it needs an action-log witness, an actual recorded sequence in which the mechanic observably fires. Human abstractions (“place,” “select,” “switch”) are exactly where phantom mechanics breed. After the incident I ran the full registry against the source narrations, every claim, both quoted; the phantom-release turned out to be the only full instance of its class, but the sweep surfaced its relatives: an assumed reversibility of an action only ever observed one-way, a debunked model surviving in a second location after the first was corrected, colors mined from one renderer asserted as fact in another.
On the trust side: provenance tiers with teeth. Every piece of knowledge the agent holds now carries a machine-stamped tier: verified by winning, confirmed on a clone, unverified hypothesis, mined from narration. A verified experiment outranks a mined prior, and when the two conflict, the stored prior goes on trial first, not the fresh evidence. If that rule sounds familiar, it is the Hallucinated Compliance countermeasure from the taxonomy, applied one level up: do not let a claim’s citation stand in for the claim’s truth, even when, especially when, the citation is your own knowledge base.
And the mining failure has a sibling that anyone building memory products should sit with: memory that outlives its truth. A stored note reading “this game is one move from cracking” survived weeks past the moment it stopped being true, and later steered real work toward a target that no longer existed. It was caught only because the machinery checks a run’s starting state against the engine instead of trusting the note. A knowledge base does not just need facts verified at write time; it needs a way for stored facts to die when the world moves.
The verification machinery
Which brings me to the half of this build I did not plan.
Agent evaluation has a quiet integrity problem. An agent run produces a narrative: I did X, I verified Y, the level is solved. Narratives are exactly what the taxonomy says they are: the place where models assert compliance instead of demonstrating it. If the program’s public claims were going to rest on agent runs, then “the agent said so” could not be evidence. So piece by piece, incident by incident, the build grew an evidence system, and at some point I realized it had become the second product of the program. What follows is the architecture of it; principles, not implementation.
Nothing counts unless a receipt says it counts. The core invariant, converged on independently by every audit round: no paid action’s result may be consumed until a single verdict function classifies it against the live game, and only a live level advance can mint a “solved” claim. Solved-ness comes from the engine, never from the model’s account of itself. Every level gained banks an immutable checkpoint: an engine-native replay log that is restored and verified exact before any future run builds on it. Identity is payload, not filename.
Runs are pre-registered. Before a paid run launches, its registration is frozen: which game, from which exact banked state, what the target is, and, the part doing the real work, a falsifier: what a miss would and would not prove. The run configuration is bound into a hermetic manifest (clean tree, frozen commit, file hashes for every load-bearing config), and every registered checkpoint is replay-validated for free before a dollar is spent. When a run ends, the target is adjudicated by machine from the run summary. The narration is not consulted.
Misses get a causal ladder, never a shrug. “Variance” is a banned explanation. Every miss is decomposed: was the knowledge even delivered to the agent? Was the delivered knowledge true? (Those are different questions; the phantom-release run passed the first and failed the second, and the ladder now checks both.) Did the executor do what it claimed? Was the target predicate itself wrong? Only what survives all of that gets attributed to the model. One recent evaluation round’s four misses turned out to be four different causes, driving four different fixes. Averaging them into a retry would have fixed none.
The instruments get audited too. The deepest lesson of the build: an instrument that chooses its own reference at measurement time measures itself. I caught a probe validating the simulator using a digest that could not see the state the simulator was faking. I caught a coverage witness “clearing” a search domain by re-deriving the domain instead of testing the production one. Both instruments were themselves corrected by the same audit process they serve, and both incidents are now permanent regression fixtures. The suspicion of the simulator promised earlier is structural for the same reason: fidelity turned out to be a property of a game state, not a game, so the agent re-runs a cheap discipline probe at every newly reached level, and a run whose simulator flips cannot validate. Which points at the process that makes all of this move:
Adversarial audit before spend. Every build change goes to independent audit lanes, deliberately different in character (one reproduces defects against the real engine and files ranked findings with patch contracts; another does fresh-eyes config and strategy review), before it is allowed to touch a paid run. The auditors’ job is to break things, and they do: one build took seven pre-launch audit cycles to clear; over a dozen real defects have been killed at zero cost before launch. The record cuts both ways, which is what makes it credible: it includes auditor findings that were rejected on evidence, corrected in place with dated markers. Every accepted defect becomes a permanent fixture in a gate that runs before every change, so the same class cannot ship twice. And the fixtures do not stop at regressions: the gate red-teams the validators themselves with coherently forged evidence. Receipts are tampered and then re-hashed so every checksum is internally consistent: ghost authority events, stripped artifact lists, relabeled outcomes, backdated build identities. The system must reject every one of them from primary evidence, not from checksums. The design rule that fell out generalizes to any evidence system: a receipt is a report, never its own proof. Every claim has to be recomputable from the underlying logs, the engine, or version control; the receipt just tells you where to look.
Negative results are enforceable artifacts. The piece I have not seen anywhere else. One game’s level resisted everything. Instead of letting it eat budget indefinitely, a stop rule was pre-registered: a search-effort threshold at which the line would be declared infeasible under the current method, written down before the evidence existed. The search ran governed, resumable, zero-cost cycles until it crossed the threshold with no win: over 151,000 validated search states. The rule tripped and minted a retirement receipt, every component of which is independently recomputable from the banked evidence. The scope is exactly as narrow as the evidence: infeasible at this entry, under this action model, by this search, not “unwinnable.” And the system now physically refuses to register a new paid run on the retired claim; the budget rerouted to the next-best target by a comparison that was also registered in advance. In a field where negative results are usually a shrug and a retry, a machine-adjudicated, skeptic-recomputable, spending-enforced “this does not work” is, I think, genuinely new.
Why the two halves need each other
Expert reasoning compilation is precisely the kind of claim this field has learned to discount. The verification machinery is what converts it from an assertion into an auditable object. Provenance separates what the agent discovered from what was decoded for it, per level, with dates. The answer-key boundary is enforced by ledger and fixture, not by promise. Wins replay exact from banked receipts. Failures carry causal classification. A skeptic does not have to trust any narrative in the stack, mine included; the evidence chain is designed to be recomputed.
And the dependency runs the other way too. The machinery exists because the compilation thesis forced honest contact with hard, cheap-to-fake results. Verification systems built in the abstract tend to check what is easy. This one was grown adversarially against a live agent that kept finding new ways to be convincingly wrong, which is why its fixture set reads like a fossil record of actual failures rather than a compliance checklist.
The HLE writeup called the failure taxonomy the program’s most durable artifact. I now think it is half of a pair. The taxonomy names the ways models are wrong; the machinery is what noticing those failures looks like when it grows teeth: every mode from the taxonomy, countered not with a reviewer’s checklist but with receipts, gates, and pre-registration around a live agent.
Results post to follow when the evaluation completes. The score will matter less than what ships with it: for every claim, a way to check.