I Built a Private Agent Eval From My Own Git History
Key Takeaways
- Both agents solved all ten tasks on every run: Claude Code with Opus 5 went 30 for 30, and Codex with GPT-5.6 Sol did the same.
- My harness failed more often than either agent. Two grader bugs and a usage limit produced every apparent failure, including a 20-point gap between the agents that didn’t exist.
- Tasks from a young repository’s first days saturate. An eval that everything passes can’t rank agents, so check that your tasks can be failed before you compare anything.
- Close the leaks (future git history, network, the original repository) and audit anyway: no transcript contained a history or network command.
Public coding benchmarks have had a rough year. OpenAI stopped reporting SWE-bench Verified in February after auditing 138 of its problems, finding material issues in 59.4% of them, and finding that every frontier model it tested could reproduce, for some tasks, the original fix or verbatim details of the problem statement. A month later, METR reported that roughly half of the test-passing SWE-bench Verified PRs it studied would not be merged by the projects’ maintainers.
Underneath both findings sits a question every team running a coding agent eventually has to answer: if the public number doesn’t tell you how an agent will do on your code, what does?
The obvious answer is to test it on your code. So I did, end to end, on a Rust project of mine: ten real commits turned into tasks, two coding agents, three runs each, sixty sandboxed runs in total. The harness is a shell script and a Python scorer. The results are below. So are the three ways my own harness misreported them before I caught it, which turned out to be the more useful finding.
Why Build a Private Eval Instead of Using a Benchmark?
A private eval measures the thing you actually deploy: an agent working on your codebase, on tasks no model could have trained on. A public benchmark measures how agents rank on someone else’s code, and it inherits that benchmark’s contamination and grading errors.
Public benchmarks aren’t useless. Newer ones built from larger tasks separate agents far more sharply than SWE-bench Verified did by the end: Specific Labs’ Real-SWE ranges from 38.8% for Fable 5.1 in Claude Code down to 16.2% for GPT-5.6 Sol in Codex CLI, averaged over eight runs per task, on tasks whose reference solutions edit a median of 11 files. That tells you a lot about frontier capability. It tells you much less about whether an agent can follow the conventions of your project, pass your tests and leave the rest of your suite alone.
A private eval answers that narrower question. It is small and noisy, and it is about the right thing.
How Do You Turn Git History Into Agent Eval Tasks?
Pick merged commits that changed both code and tests. Check out each commit’s parent, copy in the commit’s test files, and run the suite. Tests that fail there and pass on the real commit become the task’s pass criteria. The commit message becomes the prompt.
The repository is quecto, a Rust workspace I started on July 9, 2026. It has 332 commits and roughly 18,000 lines of Rust outside its test files. Here’s the pipeline.
1. Find candidates. I took 19 commits that touched both source and test files.
2. Classify them by what the tests actually check. For each candidate I ran the full workspace suite twice: on the parent with the commit’s tests copied in, and on the commit itself.
- 10 were usable. At least one test ran and failed on the parent, then passed on the commit. These are the fail-to-pass tests, and they define “solved”.
- 4 failed only at compile time. The new tests called a function or field that didn’t exist yet. That grades whether the agent guessed an API’s exact shape, not whether it built the behavior, so I dropped them.
- 3 had no signal. The new tests already passed on the parent.
- 2 were unusable for other reasons.
3. Write the prompt from the commit message. I stripped trailers and test names, then added acceptance criteria for anything the hidden tests check that the agent couldn’t discover from the code: exact help text, exit codes, environment variable names. Without that step, an agent can build the right behavior and fail on wording, and the eval ends up measuring telepathy. Here’s a complete prompt:
Implement this change in the repository in the current directory.
fix: skip leading SSE comment lines; shell-escape --init exports
- quecto_stream: skip leading `:` SSE comment/heartbeat lines when
sniffing the first data frame, so proxies that emit a comment before
the first `data:` frame don't get misrouted into the non-SSE
fallback (which would hard-fail a valid stream). Also trim the first
line before checking the `data:` prefix.
- run_init: single-quote export values with embedded-quote escaping so
`eval "$(quecto --init)"` is safe against `$`, backticks, and `"` in
values like QUECTO_SYSTEM.
Acceptance criteria:
- Export values are single-quoted, e.g. `export QUECTO_MODEL='qwen'`, and an embedded single quote is written as `'\''`.4. Grade against the whole suite. When the agent finishes, restore the real commit’s test files over whatever the agent wrote, then run every test in the workspace. A run counts as resolved only if every fail-to-pass test passes and every test that passed both before the change and on the real commit still passes. The second condition catches the agent that fixes the target by breaking something next to it. (It took me a bug to get “both” into that sentence; more on that below.)
How Do You Stop the Agent From Reading the Answer?
There are three leaks: future git history, the network, and the original repository on disk. Close all three, test that they’re closed, and then search the transcripts anyway.
Future history. An agent handed a checkout of the parent commit can still find the fix if the commit object is reachable. This has happened in the wild: on SWE-bench, agents ran git log --all and read the future commits that fixed the issue they’d been given. My setup clones the repository, checks out the parent, removes the remote, deletes every other ref, expires the reflog, prunes, and then refuses to start if the task commit is still readable.
Memorization. quecto’s first commit came after both models’ published cutoffs: a May 2026 training data cutoff for Claude Opus 5 and an April 30, 2026 knowledge cutoff for GPT-5.6 Sol. Neither model’s training data should contain these solutions. Commits dated after your model’s cutoff are the one form of contamination control you get for free.
Network and disk. Both agents ran inside their harness’s own OS sandbox, with no network and no read access to the original repository or the grading files. Claude Code used its sandbox settings with an empty domain allowlist. Codex used a permission profile with networking disabled and the same deny rules. Before any real run, I gave each agent a probe task: run the test suite, read a file from the original repository, and fetch example.com. Both builds worked and both reads and fetches failed.
The probe caught two problems. My first Claude Code configuration hid the Rust toolchain from the agent’s shell, so cargo couldn’t run at all, which makes for a very safe and entirely useless eval. And quecto’s HTTP and streaming tests, two test files, start a mock server on localhost, which Codex’s no-network mode blocks. I couldn’t get Codex’s domain allowlist to take effect from the command line (with networking switched on, it reached example.com despite the allowlist), so I blocked localhost for Claude Code as well. Neither agent could run those two test files during its work. Grading happens outside the sandbox, so it’s unaffected.
Audit. After every run the scorer searches the transcript for git log --all, reflog, fsck, the original repository’s path, GitHub URLs, and curl or wget. None of the transcripts contained any of them.
What Happened When Two Agents Ran It?
Both agents solved every task on every run. No run broke a test that passed before the change, and no transcript shows an attempt to read git history or reach the network.
Claude Code ran Opus 5 at its default settings. Codex ran GPT-5.6 Sol at medium reasoning effort. Each task ran three times per agent, one run at a time, on a fresh copy of the repository.
| Claude Code (Opus 5) | Codex (GPT-5.6 Sol) | |
|---|---|---|
| pass@1 (average over 3 runs) | 100% | 100% |
| pass^3 (solved on all 3 runs) | 10 of 10 tasks | 10 of 10 tasks |
| Median time per run | 111 seconds | 141 seconds |
| Median lines added vs. the real commit (source files) | 1.16x | 1.05x |
| Runs that also edited test files | 30 of 30 | 30 of 30 |
| Usage | $28.60 total, $0.84 median per run | 18.5M input tokens (17.1M cached), 173,000 output tokens |
The usage row isn’t comparable across the two columns. Claude Code reports a dollar estimate for each session. Codex reports tokens. Both ran on subscription plans, so neither figure is a bill.
The tasks, with the size of each real commit:
| Task | Lines added by the real commit (source files) | Claude Code | Codex |
|---|---|---|---|
| Gate project flavor privileges on trust | 92 | 3 of 3 | 3 of 3 |
| Error on an empty streaming body | 4 | 3 of 3 | 3 of 3 |
--init environment bootstrapper | 60 | 3 of 3 | 3 of 3 |
| Wire flavors and value flags into the CLI | 199 | 3 of 3 | 3 of 3 |
Document flags in --help, confirm resume | 8 | 3 of 3 | 3 of 3 |
| Interactive chat REPL | 160 | 3 of 3 | 3 of 3 |
| Scaffold new flavor manifests | 53 | 3 of 3 | 3 of 3 |
| Reject unknown leading flags | 9 | 3 of 3 | 3 of 3 |
| Skip SSE comment lines, shell-escape exports | 13 | 3 of 3 | 3 of 3 |
Load instructions, seed context, --no-verify | 19 | 3 of 3 | 3 of 3 |
That is not the scoreboard I first got.
The Eval Failed More Often Than the Agents Did
My first complete scoreboard gave Claude Code 100% and Codex 80% pass@1, with Codex solving only half the tasks on all three runs. That’s a 20-point gap, and a headline. Every one of the failures behind it was mine.
1. The parser dropped results. The very first graded run, Claude Code’s, came back unresolved: the target test passed, but two other tests were reported as broken. Both had passed. Cargo runs tests in parallel, and a neighbouring test printed output between one test’s name and its ok, so a line that should have read test approval::tests::modes_resolve_ask_safely ... ok came out with a stray ● write_file in the middle. My parser read that as a test that never finished. I fixed it by reading results from the list of failures the test runner prints at the end.
2. The grader held agents to a test they couldn’t see. Two Codex runs on the --init task solved the target but were marked as breaking init_exports_skips_blanks. That test didn’t exist before the change. The real commit added it inside a source file, and my grader only restores test files, so it was never in the agent’s copy. Codex had written its own version of the same test under a different name, and it passed. The fix is to count a test as “must still pass” only if it passed both before the change and after the real commit.
3. The agent never ran. Four Codex runs near the end of the batch exited within 26 seconds having changed nothing. The transcripts say why: the plan’s usage limit. The scorer had counted each as a failed task. I removed them, waited for the limit to reset, and reran the same four. All four resolved.
None of these would have shown up in a summary table. Each looked exactly like an agent failing a task. Three habits would have caught all of them before they reached a draft:
- Score the reference solution first. The real commit should come out resolved and the parent unresolved, with nothing else failing. If not, every agent number is noise.
- Treat a run with a nonzero exit code or an empty diff as an infrastructure failure until you’ve read its transcript.
- Read every failure before counting it. At sixty runs that’s feasible by hand. At six hundred, the first two habits have to be automated.
What Does a Perfect Score Tell You?
It tells you the tasks were too easy to separate these agents, not that the agents are equally good. An eval where everyone gets 100% has stopped measuring anything, and this one got there on its first night.
The reason is visible in the task table. All ten commits come from quecto’s first four days. The real solutions add a median of 36 lines of source code and touch one or two source files each. Real-SWE’s reference solutions edit a median of 11 files. The selection rule pushed the same way: a commit only became a task if its new tests failed at runtime on a clean parent, which favours small, self-contained changes over sprawling ones.
The eval still earned its keep. It’s a working, leak-tested harness, and pointing it at harder commits is now cheap. quecto’s later pull requests are a different size: capsules changed 12 files and added 4,265 lines, and native Anthropic API support changed 12 files and added 2,453. The four tasks I dropped for compile-only failures could come back with prompts that name the expected API. And a suite that sits at 100% is a regression alarm: if a model or harness update ever drops a run, that’s a signal worth reading.
What it can’t do yet is tell you which agent to pick. That takes tasks that some runs fail.
What Are the Limits of an Eval This Small?
- Ten tasks is a small sample. One task flipping moves pass@1 by ten points. Treat differences under that as noise.
- The tasks are small. A median of 36 source lines and one or two source files per real commit, against Real-SWE’s median of 11 files edited. That is why both agents saturated.
- Some reference solutions were written with agents. Two of the ten task commits (the
--helpand unknown-flag fixes) are the same patches as pull requests whose descriptions say “Generated with Codex”, and a third carries a Claude Opus 4.8 co-author trailer. The other seven carry no agent attribution. For those three tasks the prompt comes from a commit message an agent wrote, which could favour the agent whose style it matches. Here both agents solved all three on every run, so there was no gap for that bias to explain. - It’s my repository. I chose it before seeing any results, but a different codebase could produce a different ranking.
- Default settings. Codex ran at medium reasoning effort, and Claude Code at its defaults. Each harness might do better with tuning.
How Do You Build One for Your Own Repository?
- List commits that changed both code and tests, dated after your model’s training cutoff.
- For each, run the suite on the parent with the new tests copied in, and on the commit. Keep commits with runtime fail-to-pass tests.
- Write the prompt from the commit message. Add acceptance criteria only for strings and codes the agent can’t discover.
- Give the agent a copy at the parent with future history deleted, inside a sandbox with no network and no access to the original repository. Probe the sandbox before the first real run.
- Restore the real tests and run the whole suite. Resolved means all fail-to-pass tests pass and every test that passed both before the change and on the real commit still passes.
- Score the reference commit first. Then run every task at least three times and report both pass@1 and pass^3.
- Search the transcripts for history and network access even though the sandbox should have blocked it.
For the metrics side of this (why repeated runs, why process matters as well as outcome), see how evals should work for agentic AI.
Further Reading
- OpenAI (2026). Why OpenAI no longer evaluates on SWE-bench Verified. The audit of 138 problems and evidence of contamination.
- METR (2026). Many SWE-bench-passing PRs would not be merged. Maintainer review of test-passing agent patches.
- SWE-bench issue #465. Agents reading future commits through git history. The leak and the mitigation.
- Specific Labs (2026). Real-SWE. A benchmark of larger, real-world tasks, run eight times per agent.
- Cognition (2026). FrontierCode. Maintainer-built tasks.
More essays at Call to Think · About this project · More on this topic in the AI Evaluation collection.
Frequently Asked Questions
How do you build an eval for AI coding agents from git history?
Pick merged commits that changed both code and tests. For each one, check out the parent commit, copy in the commit's test files, and run the suite: the tests that fail there and pass on the real commit become the task's pass criteria, and the commit message becomes the prompt. After the agent works on a clean copy, restore the real test files and run the whole suite. The task is resolved only if every target test passes and nothing that used to pass now fails.
How do you stop a coding agent from cheating on a git-based eval?
Close three leaks. Delete future history from the agent's copy of the repository (remote, branches, tags and reflog), because agents have been observed running git log --all to read the fix. Run the agent in a sandbox with no network and no read access to the original repository or the grading files. Then search the transcripts for commands like git log --all, reflog or curl anyway, because a sandbox you haven't tested is only a hope.
Why not just use SWE-bench to evaluate a coding agent?
A public benchmark answers a different question: how agents rank on someone else's code. It can also be contaminated. OpenAI stopped reporting SWE-bench Verified in February 2026 after finding material issues in 59.4% of 138 audited problems and evidence that the frontier models it tested had seen some tasks. A private eval built from commits made after a model's training cutoff can't have been memorized, and it measures the agent on the code you actually care about.
What is the difference between pass@1 and pass^k for agents?
pass@1 is the fraction of tasks an agent solves on a single attempt, usually averaged over several runs. pass^k is the fraction it solves on every one of k attempts. The gap between them is how much of the score depends on luck. In this eval both agents scored 100% on both, which says the tasks were too easy to separate them, not that the agents are equally reliable.