We raised our Series A.
Here is what we are building next.Our modernization work with CMS,
18 million lines of Medicare claims code made readableWe raised our Series A.
Here is what we are building next.

Same Model, Different Harness

A 35-point swing from changing only the harness

Published:Oct 9, 2026·6 min readinResearch
Author:Arjun Goray

Most coding benchmarks rank models, but nobody actually runs a model on its own. Every coding agent pairs a model with a harness: the program that writes its prompts, hands it tools, edits files and decides what to do when a step fails. Teams pick the two together, yet leaderboards rarely show how much the harness matters.

So we measured it. 8090 Labs, our new research team, ran Datacurve’s DeepSWE v1.1 on seven models in five popular harnesses. Every model ran every task twice in each harness, and Datacurve’s own tests graded every run.

Benchmark: Datacurve DeepSWE v1.1
  • 113 real coding tasks
Models tested: 7*
  • GPT-5.6 Sol
  • GPT-5.6 Terra
  • GPT-5.6 Luna
  • Claude Fable 5.1
  • Grok 4.6
  • GLM 5.3 Flash
  • DeepSeek V4 Flash 0731
Harnesses tested: 5
  • Codex
  • Claude Code
  • OpenCode
  • Pi
  • DeepSeek harness

* This is not a leaderboard of models or harnesses. New models ship almost weekly, so any lineup trails the latest model or harness releases, and some popular models (Opus, Sonnet, Astra, Kimi K3) aren’t in this run. The point is the pairing. We’ll add models and shorten the time from run to publication in future releases.

The harness can move a model 35 points, or barely at all

We kept each model and its reasoning setting fixed and changed only the harness. In the worst case, GLM 5.3 Flash’s score varied by 35 points between its best and worst harness: it passed 24% of its runs in Codex and 59% in the DeepSeek harness. GPT-5.6 Sol, on the other hand, stayed steady whichever harness it was in, moving just 6 points.

Most models landed somewhere in between. Claude Fable 5.1 moved 20 points and DeepSeek V4 Flash 0731 moved 17, while Grok 4.6 stayed within 5.

Figure 1. Every model’s best and worst harness
Models ordered by their best resultClick a mark to see its runs
CodexClaude CodeOpenCodePiDeepSeek harness
Each row is one model and each mark is one harness. The number at right is the gap between its best and worst harness; gaps of 9 points or less are grey because they’re negligible.

The harness can matter more than the model, too. GPT-5.6 Sol, the strongest model we tested, beat GLM’s best result by only 16 points.

No model did best in its maker’s own harness

Three of the five harnesses come from model makers: Codex from OpenAI, Claude Code from Anthropic and the DeepSeek harness from DeepSeek. You’d expect each model to do best in its maker’s own harness, but none of the five did.

Figure 2. Each model in its maker’s harness against its best harness
Click a mark to see its runs
The hollow mark is the harness built by the model’s maker and the filled mark is the model’s best harness. The number at right is the gap between them; gaps of 9 points or less are grey because they’re negligible.

For the GPT-5.6 models and DeepSeek V4 Flash 0731, the gap is negligible. Claude Fable 5.1 is the exception: it passed 41% of its runs in Claude Code but 61% in Codex. Its Claude Code runs were also much shorter, with a median of 28 tool calls, whereas in Codex the median was 66.

Passing once is not passing reliably

A success rate averages over runs, so it can’t tell you whether a model will pass the same task again. We ran every task twice in each harness and counted how often both runs passed. In the DeepSeek harness, GLM passed both runs on 51 of 113 tasks; in Codex, it managed that on only 16.

Figure 3. How many tasks passed in both runs, one run or neither
Tasks with two graded runs
Each task was run twice in every pairing. Bars count tasks by how many of the two runs passed.

Two runs per task is a small sample, so treat these counts as an upper bound on reliability. If you’re running agents in production, repeatability matters as much as the average.

Cost per success tells a different story

The price of a run is only part of what you pay, because failed runs cost money too. For each pairing, we divided the total API cost of all its runs by the number that passed, which gives the cost per success.

GLM’s runs were cheapest in Codex, at $0.22 each, but only 24% of them passed, so each success cost $0.93. In Pi, its runs cost a little more at $0.25, but 58% passed, which brought the cost per success down to $0.43.

Figure 4. Cost per run and cost per success in each harness
Cost per success is total API spend divided by the runs that passed, at each model maker’s list rates.

Cheap tokens aren’t the same as cheap work. The harness with the cheapest runs cost more than twice as much per success.

The best harness can change with the language

Every harness ran the same tasks in each language, so a gap between two harnesses on one language comes from the pairing, not from how hard the tasks are. We break out Go, Python and TypeScript, which make up 103 of the 113 tasks. Claude Fable 5.1 shows it most clearly. In OpenCode, it was within 3 points of its best harness on Go but 20 points behind on TypeScript.

Figure 5. Success rate by language in each harness
In each group the bars are Go, Python and TypeScript, top to bottom. Each has about 34 tasks, so differences under about 17 points could easily be chance. Rust and JavaScript have only 5 tasks each, too few to chart; they count toward every overall figure.

What this means for teams choosing an agent

The harness is part of a model’s performance. A leaderboard score for the model alone hides how much its harness adds or takes away, and that effect isn’t the same from one model to the next.

For teams choosing an agent, a few things follow:

  • Test the model and harness together. A model that looks weak in one harness can hold its own in another.
  • Check how often a pass repeats. An average can hide a model that solves a task once and then fails it.
  • Price the work by cost per success. The cheapest run isn’t always the cheapest result.

Every run behind these results, with the agent’s full trajectory, is in the run inspector.

Appendix: full results

Each cell is one model in one harness. You can switch between success rate and cost per success, and select any cell to see the runs behind it.

Figure 6. Every model in every harness
Share of graded runs that passed. Brighter cells are higher, and each model’s best harness is outlined.

Appendix: how we measured

Tasks
Datacurve DeepSWE v1.1: 113 real bug fixes and features from open-source Go, Python, TypeScript, Rust and JavaScript projects.
Harnesses
Codex, Claude Code, OpenCode, Pi and the DeepSeek harness. Each harness was connected to Harbor as officially recommended.
Models
Two open-weight models (GLM 5.3 Flash, DeepSeek V4 Flash 0731) and popular closed source models at the time. Reasoning was set to max for GPT-5.6 and Fable 5.1, xhigh for Grok 4.6 and high for GLM and DeepSeek.
Providers
Claude and GPT-5.6 models ran through the Anthropic and OpenAI APIs or Amazon Bedrock. The open-weight models ran through a mix of Fireworks and Baseten to stay within rate limits. Grok 4.6 ran through the xAI API.
Runs
We ran every task twice for each model and harness, each time on its own isolated runner.
Grading
Datacurve’s own tests. A run only counts if it passes them.
Noise
Each pairing has about 226 runs. At that size, a gap of 9 points or less between two pairings is within run-to-run noise (95% interval), which is why we call those gaps negligible. A pairing’s first and second runs on the same tasks differed by 4 points on average. Results for a single language use only about 34 tasks, so their noise is closer to 17 points.
Cost
Recorded token counts multiplied by each model maker’s official list rates, whichever provider served the run.
Data
Per-task results, prompts and full agent trajectories are public in the run inspector or as JSON.