Practical use and limits
Use it for: Start with real issues, freeze the repository snapshot and acceptance tests, and record final diff quality, retries, tool errors, time, cost, and human cleanup.
Limits: A small benchmark cannot represent every workload. Avoid answer leakage, repeat stochastic runs, and define failure labels before broad claims.
Start from real work
Collect twenty to fifty representative tasks from issue trackers, support conversations, and recent pull requests. Remove secrets and define a clean starting snapshot. Include easy, medium, and uncomfortable tasks: ambiguous requirements, incomplete tests, flaky commands, and changes that should be rejected. The goal is not to make a model look good; it is to learn where an agent can safely save time.
Define success before the run
For each task, write the acceptance test, allowed files, required commands, expected side effects, and a stop condition. A task is successful only when the final artifact meets those rules. A fluent explanation without a correct patch is a failure. If the task is subjective, define a short human rubric before looking at model outputs.
Keep the harness constant
Use the same system instructions, repository context policy, tool schemas, sandbox permissions, timeout, retry budget, and verifier for every model. Log the model identifier and all configuration values. If you change the harness while comparing models, you are measuring a combined system change and should label it that way.
Track more than pass or fail
Record whether the model completed the task, how many tool calls it made, how often it produced invalid arguments, how long it took, how many tokens it used, whether it needed a retry, and how much human cleanup remained. Add labels for common failures such as wrong file selection, premature success, context loss, unsafe command, and test misinterpretation. These labels tell you what to improve next.
Use a rubric that separates correctness from safety
Score each run on distinct dimensions instead of collapsing everything into one impression. A practical rubric can assign points for functional correctness, adherence to scope, verification quality, explanation, and preservation of unrelated work, with an automatic zero for leaked secrets or unauthorized destructive actions. Have two reviewers independently grade a small overlapping sample and resolve disagreements before scoring the full set. If reviewers cannot apply the rubric consistently, the benchmark is not yet measuring a stable outcome.
Control variance, contamination, and leakage
Run important tasks more than once with the same settings and report the number of attempts, not only the best trace. Keep hidden acceptance tests outside the model context, rotate equivalent task variants, and quarantine tasks that may have appeared in public training or vendor demos. Compare results from the same repository snapshot and reset all generated files between runs. These controls will not remove uncertainty, but they make it harder for luck, cached context, or benchmark familiarity to masquerade as capability.
Publish failure cases
A credible evaluation includes representative failures and the exact reason they failed. Keep traces redacted, but preserve enough information to reproduce the result. When a benchmark result changes, record whether the cause was a model update, prompt change, tool implementation, dependency update, or task leakage. This turns a one-time comparison into an engineering instrument.
Frequently asked questions
How many tasks are enough?
Start with a small set that reflects your real work, then expand it when a result would change a decision. Twenty carefully designed tasks are more useful than hundreds of generic prompts with unclear grading.
Should I report the best run or the average?
Report the full protocol and distribution: number of attempts, completion rate, median cost and latency, and notable failures. A best run can demonstrate possibility, but it does not estimate reliability.