SWE-bench Verified · 500 instances · one attempt each
The full SWE-bench Verified set — 500 real GitHub issues from twelve Python repositories — run end to end through Benzi on DeepSeek V4.1-Flash, graded by the official SWE-bench harness inside its own per-instance Docker images. One attempt per instance.
swebench.harness.run_evaluation
Why cost is the number to look at. SWE-bench's leaderboard ranks on resolve rate alone and publishes nothing about what a run costs. The whole 500-instance run came to $9.26 — under ten dollars for a complete pass at a benchmark where published submissions routinely spend orders of magnitude more.
We ran the same 500 instances through mini-swe-agent, the
SWE-bench team's own minimal reference agent (version 2.4.6, installed from PyPI; source at
github.com/SWE-agent/mini-swe-agent), on the same model. It resolved 410 of 500 (82.0%) and read
645,752 lines; Benzi read 326,928, which is 49.4% fewer. Lines read are every line of output from a
command that views text (cat, sed -n, head, tail,
nl), counted the same way for both agents, plus Benzi's read_source lines.
mini-swe-agent run details
Grading. Official SWE-bench per-instance Docker images and
swebench.harness.run_evaluation (swebench 5.0.2) against the official
SWE-bench/SWE-bench_Verified dataset. A patch counts only if every FAIL_TO_PASS test
passes and no PASS_TO_PASS test regresses.
No network. GitHub and PyPI hosts resolved to 127.0.0.1 inside every container, so nothing could fetch the fixed version of itself.
Retries were for infrastructure only. Instances hit by Docker, API or other infrastructure failures were re-run. No instance was retried for being wrong; every instance is reported from its one attempt.
30–31 October 2026
Every instance in the split, grouped by the repository the issue came from.
| Repository | Instances | Resolved | Rate |
|---|---|---|---|
| flask | 1 | 1 | 100.0% |
| scikit-learn | 32 | 31 | 96.9% |
| pytest | 19 | 17 | 89.5% |
| django | 231 | 198 | 85.7% |
| matplotlib | 34 | 28 | 82.4% |
| xarray | 22 | 18 | 81.8% |
| sympy | 75 | 59 | 78.7% |
| sphinx | 44 | 34 | 77.3% |
| astropy | 22 | 17 | 77.3% |
| pylint | 10 | 5 | 50.0% |
| requests | 8 | 4 | 50.0% |
| seaborn | 2 | 1 | 50.0% |
All 500 runs, including the ones we lose. Click a column to sort.
| Instance | Repo | Result | Turns | Cost ($) |
|---|
Cost at DeepSeek V4.1-Flash standard (off-peak) list prices: $0.15/M input, $0.003/M cached input, $0.60/M output.
14 October 2026
Every instance in the split, grouped by the repository the issue came from.
| Repository | Instances | Resolved | Rate |
|---|---|---|---|
| seaborn | 2 | 2 | 100.0% |
| flask | 1 | 1 | 100.0% |
| pytest | 19 | 18 | 94.7% |
| scikit-learn | 32 | 30 | 93.8% |
| requests | 8 | 7 | 87.5% |
| xarray | 22 | 19 | 86.4% |
| django | 231 | 197 | 85.3% |
| sympy | 75 | 58 | 77.3% |
| matplotlib | 34 | 26 | 76.5% |
| sphinx | 44 | 33 | 75.0% |
| astropy | 22 | 15 | 68.2% |
| pylint | 10 | 6 | 60.0% |
All 500 runs, including the ones we lose. Click a column to sort.
| Instance | Repo | Result | Turns | Cost ($) |
|---|
Cost at DeepSeek V4.1-Flash standard (off-peak) list prices: $0.15/M input, $0.003/M cached input, $0.60/M output.
15–16 August 2026
Every instance in the split, grouped by the repository the issue came from.
| Repository | Instances | Resolved | Rate |
|---|---|---|---|
| requests | 8 | 8 | 100.0% |
| flask | 1 | 1 | 100.0% |
| pytest | 19 | 18 | 94.7% |
| scikit-learn | 32 | 29 | 90.6% |
| django | 231 | 179 | 77.5% |
| sympy | 75 | 58 | 77.3% |
| xarray | 22 | 17 | 77.3% |
| matplotlib | 34 | 26 | 76.5% |
| sphinx | 44 | 31 | 70.5% |
| astropy | 22 | 15 | 68.2% |
| seaborn | 2 | 1 | 50.0% |
| pylint | 10 | 4 | 40.0% |
All 500 runs, including the ones we lose. Click a column to sort.
| Instance | Repo | Result | Turns | Lines read | Wall (s) | Cost ($) |
|---|
Grading. Official SWE-bench per-instance Docker images and
swebench.harness.run_evaluation. A patch counts only if every FAIL_TO_PASS test
passes and no PASS_TO_PASS test regresses.
No network. github.com, raw.githubusercontent.com, api.github.com, codeload.github.com, objects.githubusercontent.com, gist.github.com, pypi.org and files.pythonhosted.org all resolved to 127.0.0.1 inside every container. Across the run the agent made 244 egress attempts; none returned remote content.
No timeouts. No wall-clock cap was applied to any instance, so a slow solve is never discarded as a failure.
On the wall times. They should not be read as a measure of how fast the agent is, and they are not comparable with the timings on the 24-bug benchmark — that ran on different hardware entirely. They are also only loosely comparable with each other, because this run was itself split across two different machines, and instances that happened to start at the same moment contended for the same cores. An identical task could therefore take 45 minutes in one slot and 10 in another. None of it touches the result. SWE-bench scores whether the patch makes the tests pass; it is untimed, every instance ran to completion with no cap, and grading used the identical official harness throughout. Turns, tokens, lines read and cost are hardware-independent and comparable everywhere on this page.
Three instances produced no patch. On
django-13513, matplotlib-14623 and sympy-23413 the agent
investigated and stopped without attempting an edit. They are counted as failures.
Seven instances were generated twice. Each had its first run destroyed by a fault in our harness rather than by the agent — a Docker engine disconnect, a stdout-marker collision, and two dropped API connections. Instances were re-run on the basis of the fault, never on the basis of the result, and every re-run was kept as produced.
Everything run since the 387/500 result above, recorded in full. These are experiments, not submissions: each re-attempts instances the main run already answered, under changed configurations, so none of these rows can be combined with the 387 — SWE-bench scores one attempt per instance per system. They are published for the same reason as everything else on this page: check the working.
Per-run wall/turns/lines are shown where the run records survived; earlier experiment machines were shut down before their logs were archived, so those rows carry verdicts only. Per-run cost was not recomputed for experiments.
The failures the main run missed where the leading submission succeeded, re-attempted with the flagship reasoner (two never completed and are excluded). 55% converted — but a later control showed ~30% of failures convert on a plain same-model re-run, so this is an upper bound on the model-tier effect, not a measurement of it.
| instance | repo | model | wall | turns | lines read | resolved | cost |
|---|---|---|---|---|---|---|---|
| astropy-14365 | astropy | deepseek-pro | — | — | — | no | — |
| django-11149 | django | deepseek-pro | — | — | — | yes | — |
| django-11239 | django | deepseek-pro | — | — | — | yes | — |
| django-11532 | django | deepseek-pro | — | — | — | no | — |
| django-11848 | django | deepseek-pro | — | — | — | no | — |
| django-11885 | django | deepseek-pro | — | — | — | yes | — |
| django-12273 | django | deepseek-pro | — | — | — | no | — |
| django-12774 | django | deepseek-pro | — | — | — | yes | — |
| django-13297 | django | deepseek-pro | — | — | — | yes | — |
| django-13344 | django | deepseek-pro | — | — | — | yes | — |
| django-13406 | django | deepseek-pro | — | — | — | yes | — |
| django-14376 | django | deepseek-pro | — | — | — | no | — |
| django-14534 | django | deepseek-pro | — | — | — | no | — |
| django-14559 | django | deepseek-pro | — | — | — | yes | — |
| django-15037 | django | deepseek-pro | — | — | — | yes | — |
| django-15127 | django | deepseek-pro | — | — | — | no | — |
| django-15916 | django | deepseek-pro | — | — | — | no | — |
| django-15957 | django | deepseek-pro | — | — | — | yes | — |
| django-15973 | django | deepseek-pro | — | — | — | yes | — |
| django-15987 | django | deepseek-pro | — | — | — | yes | — |
| django-16454 | django | deepseek-pro | — | — | — | yes | — |
| matplotlib-14623 | matplotlib | deepseek-pro | — | — | — | yes | — |
| matplotlib-24870 | matplotlib | deepseek-pro | — | — | — | no | — |
| xarray-3993 | xarray | deepseek-pro | — | — | — | yes | — |
| xarray-4687 | xarray | deepseek-pro | — | — | — | yes | — |
| xarray-6938 | xarray | deepseek-pro | — | — | — | no | — |
| pytest-10356 | pytest | deepseek-pro | — | — | — | no | — |
| scikit-learn-14710 | scikit-learn | deepseek-pro | — | — | — | yes | — |
| sphinx-10466 | sphinx | deepseek-pro | — | — | — | yes | — |
| sphinx-8056 | sphinx | deepseek-pro | — | — | — | no | — |
| sphinx-8638 | sphinx | deepseek-pro | — | — | — | no | — |
| sympy-16886 | sympy | deepseek-pro | — | — | — | no | — |
| sympy-20916 | sympy | deepseek-pro | — | — | — | no | — |
Twenty instances (ten the main run solved, ten it failed) re-run with an upgrade_to_pro tool the agent could call if it judged more reasoning would help. It was offered in the tool list but never announced in the prompt: zero calls in twenty runs. A capability outside the model’s habitual reach behaves as if it does not exist. Four of the ten previously-solved rows failing below are grading flakes: those suites call a live external service that returned 503s that evening.
| instance | repo | model | wall | turns | lines read | resolved | cost |
|---|---|---|---|---|---|---|---|
| astropy-14598 | astropy | deepseek | — | — | — | no | — |
| astropy-7606 | astropy | deepseek | — | — | — | yes | — |
| django-11477 | django | deepseek | — | — | — | no | — |
| django-12325 | django | deepseek | — | — | — | no | — |
| django-13195 | django | deepseek | — | — | — | no | — |
| django-14140 | django | deepseek | — | — | — | yes | — |
| django-14170 | django | deepseek | — | — | — | no | — |
| requests-1724 | psf | deepseek | — | — | — | no | — |
| requests-1766 | psf | deepseek | — | — | — | no | — |
| requests-1921 | psf | deepseek | — | — | — | no | — |
| requests-2317 | psf | deepseek | — | — | — | no | — |
| requests-2931 | psf | deepseek | — | — | — | yes | — |
| requests-5414 | psf | deepseek | — | — | — | yes | — |
| xarray-4629 | xarray | deepseek | — | — | — | yes | — |
| pylint-8898 | pylint | deepseek | — | — | — | yes | — |
| pytest-5809 | pytest | deepseek | — | — | — | yes | — |
| pytest-7521 | pytest | deepseek | — | — | — | yes | — |
| pytest-7982 | pytest | deepseek | — | — | — | yes | — |
| sphinx-9602 | sphinx | deepseek | — | — | — | no | — |
| sympy-18199 | sympy | deepseek | — | — | — | no | — |
The start of a full 500-instance pro run, halted once its two questions were answered: pro converts only ~13% of the hard pile (instances no leading submission solves), and retains only ~90% of the main run’s wins — scaling the model is roughly a wash at several times the price. Rows here exclude the 33 probe instances above, which seeded this run’s ledger.
| instance | repo | model | wall | turns | lines read | resolved | cost |
|---|---|---|---|---|---|---|---|
| astropy-12907 | astropy | deepseek-pro | — | — | — | yes | — |
| astropy-13236 | astropy | deepseek-pro | — | — | — | no | — |
| astropy-13398 | astropy | deepseek-pro | — | — | — | no | — |
| astropy-13453 | astropy | deepseek-pro | — | — | — | yes | — |
| astropy-13579 | astropy | deepseek-pro | — | — | — | yes | — |
| astropy-13977 | astropy | deepseek-pro | — | — | — | no | — |
| astropy-14096 | astropy | deepseek-pro | — | — | — | yes | — |
| astropy-14309 | astropy | deepseek-pro | — | — | — | yes | — |
| astropy-14369 | astropy | deepseek-pro | — | — | — | no | — |
| astropy-14508 | astropy | deepseek-pro | — | — | — | yes | — |
| astropy-14539 | astropy | deepseek-pro | — | — | — | yes | — |
| astropy-14598 | astropy | deepseek-pro | — | — | — | no | — |
| astropy-14995 | astropy | deepseek-pro | — | — | — | yes | — |
| astropy-7166 | astropy | deepseek-pro | — | — | — | yes | — |
| astropy-7336 | astropy | deepseek-pro | — | — | — | yes | — |
| astropy-7606 | astropy | deepseek-pro | — | — | — | yes | — |
| astropy-7671 | astropy | deepseek-pro | — | — | — | yes | — |
| astropy-8707 | astropy | deepseek-pro | — | — | — | yes | — |
| astropy-8872 | astropy | deepseek-pro | — | — | — | yes | — |
| django-10097 | django | deepseek-pro | — | — | — | no | — |
| django-10554 | django | deepseek-pro | — | — | — | no | — |
| django-10880 | django | deepseek-pro | — | — | — | yes | — |
| django-10914 | django | deepseek-pro | — | — | — | yes | — |
| django-10973 | django | deepseek-pro | — | — | — | no | — |
| django-10999 | django | deepseek-pro | — | — | — | no | — |
| django-11066 | django | deepseek-pro | — | — | — | yes | — |
| django-11087 | django | deepseek-pro | — | — | — | yes | — |
| django-11095 | django | deepseek-pro | — | — | — | yes | — |
| django-11099 | django | deepseek-pro | — | — | — | yes | — |
| django-11119 | django | deepseek-pro | — | — | — | yes | — |
| django-11133 | django | deepseek-pro | — | — | — | yes | — |
| django-11138 | django | deepseek-pro | — | — | — | yes | — |
| django-11141 | django | deepseek-pro | — | — | — | no | — |
| django-11163 | django | deepseek-pro | — | — | — | yes | — |
| django-11179 | django | deepseek-pro | — | — | — | yes | — |
| django-11206 | django | deepseek-pro | — | — | — | yes | — |
| django-11211 | django | deepseek-pro | — | — | — | yes | — |
| django-11265 | django | deepseek-pro | — | — | — | yes | — |
| django-11276 | django | deepseek-pro | — | — | — | yes | — |
| django-11292 | django | deepseek-pro | — | — | — | yes | — |
| django-11299 | django | deepseek-pro | — | — | — | yes | — |
| django-11333 | django | deepseek-pro | — | — | — | yes | — |
| django-11400 | django | deepseek-pro | — | — | — | no | — |
| django-11433 | django | deepseek-pro | — | — | — | yes | — |
| django-11451 | django | deepseek-pro | — | — | — | yes | — |
| django-11477 | django | deepseek-pro | — | — | — | yes | — |
| django-11490 | django | deepseek-pro | — | — | — | yes | — |
| django-11551 | django | deepseek-pro | — | — | — | yes | — |
| django-11555 | django | deepseek-pro | — | — | — | yes | — |
| django-11603 | django | deepseek-pro | — | — | — | yes | — |
| django-11728 | django | deepseek-pro | — | — | — | no | — |
| django-11734 | django | deepseek-pro | — | — | — | no | — |
| django-11740 | django | deepseek-pro | — | — | — | yes | — |
| django-11749 | django | deepseek-pro | — | — | — | yes | — |
| django-11790 | django | deepseek-pro | — | — | — | yes | — |
| django-11815 | django | deepseek-pro | — | — | — | yes | — |
| django-11820 | django | deepseek-pro | — | — | — | no | — |
| django-11880 | django | deepseek-pro | — | — | — | yes | — |
| django-11951 | django | deepseek-pro | — | — | — | yes | — |
| django-11964 | django | deepseek-pro | — | — | — | no | — |
| django-11999 | django | deepseek-pro | — | — | — | yes | — |
| django-12039 | django | deepseek-pro | — | — | — | yes | — |
| django-12050 | django | deepseek-pro | — | — | — | yes | — |
| django-12143 | django | deepseek-pro | — | — | — | yes | — |
| django-12155 | django | deepseek-pro | — | — | — | yes | — |
| django-12193 | django | deepseek-pro | — | — | — | no | — |
| django-12262 | django | deepseek-pro | — | — | — | yes | — |
| django-12276 | django | deepseek-pro | — | — | — | yes | — |
| django-12308 | django | deepseek-pro | — | — | — | no | — |
| django-12325 | django | deepseek-pro | — | — | — | no | — |
| django-12406 | django | deepseek-pro | — | — | — | no | — |
| django-12708 | django | deepseek-pro | — | — | — | yes | — |
| django-12713 | django | deepseek-pro | — | — | — | yes | — |
| django-12741 | django | deepseek-pro | — | — | — | yes | — |
| django-13195 | django | deepseek-pro | — | — | — | no | — |
| django-13212 | django | deepseek-pro | — | — | — | no | — |
| django-13794 | django | deepseek-pro | — | — | — | no | — |
| django-14011 | django | deepseek-pro | — | — | — | no | — |
| django-14034 | django | deepseek-pro | — | — | — | no | — |
| django-14140 | django | deepseek-pro | — | — | — | yes | — |
| django-14155 | django | deepseek-pro | — | — | — | no | — |
| django-14170 | django | deepseek-pro | — | — | — | no | — |
| django-14315 | django | deepseek-pro | — | — | — | no | — |
| django-15629 | django | deepseek-pro | — | — | — | no | — |
| django-15695 | django | deepseek-pro | — | — | — | no | — |
| django-16256 | django | deepseek-pro | — | — | — | no | — |
| django-16631 | django | deepseek-pro | — | — | — | no | — |
| django-16667 | django | deepseek-pro | — | — | — | no | — |
| matplotlib-21568 | matplotlib | deepseek-pro | — | — | — | no | — |
The failure analysis showed the agent never consults its relational tools before editing, so the harness was changed to attach the profile of every edited scope to the first edit and the commit gate. On the 98 failures whose shape that information addresses it converted 13/69 attempted (19%) — under the ~30% a plain re-run converts by variance. The mechanism was removed. Published because a measured null is worth more than a plausible mechanism: information at the right moment was not the binding constraint.
| instance | repo | model | wall | turns | lines read | resolved | cost |
|---|---|---|---|---|---|---|---|
| astropy-13033 | astropy | deepseek | 1,266s | 18 | 228 | no | — |
| astropy-13398 | astropy | deepseek | 1,713s | 39 | 822 | no | — |
| astropy-13977 | astropy | deepseek | 1,587s | 46 | 703 | no | — |
| astropy-14182 | astropy | deepseek | 1,338s | 31 | 417 | yes | — |
| astropy-14365 | astropy | deepseek | 1,243s | 19 | 446 | yes | — |
| astropy-14369 | astropy | deepseek | 1,574s | 34 | 970 | no | — |
| astropy-14598 | astropy | deepseek | 1,663s | 38 | 720 | no | — |
| astropy-7606 | astropy | deepseek | 1,160s | 16 | 135 | yes | — |
| django-10097 | django | deepseek | 1,405s | 38 | 257 | no | — |
| django-10554 | django | deepseek | 3,202s | 81 | 1,247 | no | — |
| django-10973 | django | deepseek | 1,159s | 15 | 222 | no | — |
| django-10999 | django | deepseek | 1,027s | 10 | 42 | no | — |
| django-11087 | django | deepseek | 1,153s | 67 | 1,198 | no | — |
| django-11141 | django | deepseek | 655s | 38 | 323 | no | — |
| django-11149 | django | deepseek | 560s | 36 | 647 | yes | — |
| django-11239 | django | deepseek | 376s | 22 | 409 | no | — |
| django-11532 | django | deepseek | 340s | 23 | 353 | no | — |
| django-11734 | django | deepseek | 1,099s | 74 | 618 | no | — |
| django-11820 | django | deepseek | 270s | 19 | 245 | no | — |
| django-11848 | django | deepseek | 197s | 13 | 156 | no | — |
| django-11885 | django | deepseek | 1,298s | 42 | 769 | no | — |
| django-12273 | django | deepseek | 711s | 58 | 588 | no | — |
| django-12325 | django | deepseek | 447s | 25 | 193 | no | — |
| django-12406 | django | deepseek | 545s | 23 | 384 | yes | — |
| django-12774 | django | deepseek | 292s | 17 | 156 | yes | — |
| django-13195 | django | deepseek | 267s | 20 | 107 | no | — |
| django-13212 | django | deepseek | 364s | 30 | 624 | no | — |
| django-13297 | django | deepseek | 461s | 23 | 255 | no | — |
| django-13344 | django | deepseek | 1,066s | 58 | 718 | yes | — |
| django-13406 | django | deepseek | 1,103s | 66 | 1,373 | no | — |
| django-14011 | django | deepseek | 662s | 44 | 364 | no | — |
| django-14140 | django | deepseek | 263s | 19 | 364 | no | — |
| django-14155 | django | deepseek | 425s | 27 | 261 | no | — |
| django-14315 | django | deepseek | 195s | 23 | 171 | no | — |
| django-14376 | django | deepseek | 291s | 24 | 297 | no | — |
| django-14534 | django | deepseek | 622s | 34 | 149 | no | — |
| django-14559 | django | deepseek | 210s | 15 | 322 | no | — |
| django-15037 | django | deepseek | 431s | 34 | 530 | no | — |
| django-15098 | django | deepseek | 854s | 51 | 925 | no | — |
| django-15563 | django | deepseek | 787s | 43 | 473 | yes | — |
| django-15629 | django | deepseek | 752s | 54 | 819 | no | — |
| django-15695 | django | deepseek | 630s | 57 | 597 | no | — |
| django-15916 | django | deepseek | 250s | 16 | 346 | no | — |
| django-15957 | django | deepseek | 1,076s | 60 | 806 | no | — |
| django-15987 | django | deepseek | 545s | 33 | 218 | yes | — |
| django-16256 | django | deepseek | 558s | 41 | 1,698 | yes | — |
| django-16263 | django | deepseek | 1,646s | 130 | 1,520 | no | — |
| django-16454 | django | deepseek | 449s | 28 | 298 | no | — |
| django-16502 | django | deepseek | 752s | 28 | 339 | no | — |
| django-16631 | django | deepseek | 718s | 31 | 368 | no | — |
| django-16667 | django | deepseek | 300s | 19 | 368 | no | — |
| matplotlib-21568 | matplotlib | deepseek | 1,169s | 69 | 263 | no | — |
| matplotlib-23299 | matplotlib | deepseek | 530s | 26 | 555 | no | — |
| matplotlib-24870 | matplotlib | deepseek | 550s | 45 | 557 | yes | — |
| matplotlib-26208 | matplotlib | deepseek | 911s | 51 | 621 | no | — |
| matplotlib-26466 | matplotlib | deepseek | 484s | 40 | 822 | no | — |
| seaborn-3187 | seaborn | deepseek | 730s | 49 | 939 | no | — |
| xarray-3993 | xarray | deepseek | 305s | 25 | 318 | yes | — |
| xarray-4687 | xarray | deepseek | 389s | 39 | 958 | no | — |
| xarray-6938 | xarray | deepseek | 347s | 35 | 601 | no | — |
| xarray-6992 | xarray | deepseek | 198s | 22 | 274 | no | — |
| xarray-7229 | xarray | deepseek | 600s | 45 | 840 | no | — |
| pylint-4551 | pylint | deepseek | 851s | 101 | 1,105 | no | — |
| pylint-4604 | pylint | deepseek | 109s | 13 | 195 | no | — |
| pylint-4661 | pylint | deepseek | 291s | 16 | 173 | no | — |
| pylint-8898 | pylint | deepseek | 454s | 43 | 539 | no | — |
| pytest-10356 | pytest | deepseek | 421s | 36 | 562 | no | — |
| sphinx-10466 | sphinx | deepseek | 305s | 33 | 372 | yes | — |
| sphinx-7462 | sphinx | deepseek | 437s | 24 | 124 | no | — |
The same configuration re-attempting instances the main run already solved, stopped by choice at 67 graded: 63/67 held (94%). The losses cluster on instances the pro run also lost — original wins that pass roughly one attempt in three. Every re-run of any agent pays this tax; it is the empirical reason the way up is engine work, not re-rolls.
| instance | repo | model | wall | turns | lines read | resolved | cost |
|---|---|---|---|---|---|---|---|
| astropy-12907 | astropy | deepseek | 435s | 16 | 493 | yes | — |
| astropy-13236 | astropy | deepseek | 666s | 34 | 277 | no | — |
| astropy-13453 | astropy | deepseek | 480s | 29 | 782 | yes | — |
| astropy-13579 | astropy | deepseek | 822s | 31 | 1,078 | yes | — |
| astropy-14096 | astropy | deepseek | 673s | 23 | 142 | yes | — |
| astropy-14309 | astropy | deepseek | 389s | 19 | 208 | yes | — |
| astropy-14508 | astropy | deepseek | 643s | 28 | 232 | yes | — |
| astropy-14539 | astropy | deepseek | 481s | 24 | 202 | yes | — |
| astropy-14995 | astropy | deepseek | 330s | 17 | 331 | yes | — |
| astropy-7166 | astropy | deepseek | 597s | 38 | 193 | yes | — |
| astropy-7336 | astropy | deepseek | 354s | 12 | 65 | yes | — |
| astropy-7671 | astropy | deepseek | 409s | 15 | 105 | yes | — |
| astropy-8707 | astropy | deepseek | 529s | 29 | 597 | yes | — |
| astropy-8872 | astropy | deepseek | 501s | 23 | 262 | yes | — |
| django-10880 | django | deepseek | 326s | 22 | 389 | yes | — |
| django-10914 | django | deepseek | 233s | 15 | 185 | yes | — |
| django-11066 | django | deepseek | 106s | 6 | 134 | yes | — |
| django-11095 | django | deepseek | 169s | 12 | 77 | yes | — |
| django-11099 | django | deepseek | 137s | 11 | 50 | yes | — |
| django-11119 | django | deepseek | 180s | 18 | 66 | yes | — |
| django-11133 | django | deepseek | 212s | 21 | 315 | yes | — |
| django-11138 | django | deepseek | 593s | 43 | 953 | yes | — |
| django-11163 | django | deepseek | 223s | 23 | 124 | yes | — |
| django-11179 | django | deepseek | 173s | 9 | 75 | yes | — |
| django-11206 | django | deepseek | 464s | 22 | 309 | yes | — |
| django-11211 | django | deepseek | 458s | 33 | 422 | yes | — |
| django-11265 | django | deepseek | 502s | 42 | 1,177 | no | — |
| django-11276 | django | deepseek | 294s | 22 | 153 | yes | — |
| django-11292 | django | deepseek | 340s | 30 | 720 | yes | — |
| django-11299 | django | deepseek | 234s | 28 | 211 | yes | — |
| django-11333 | django | deepseek | 253s | 18 | 147 | yes | — |
| django-11400 | django | deepseek | 418s | 29 | 371 | no | — |
| django-11433 | django | deepseek | 374s | 22 | 227 | yes | — |
| django-11451 | django | deepseek | 165s | 13 | 196 | yes | — |
| django-11490 | django | deepseek | 413s | 32 | 648 | yes | — |
| django-11551 | django | deepseek | 243s | 20 | 108 | yes | — |
| django-11555 | django | deepseek | 630s | 33 | 369 | yes | — |
| django-11603 | django | deepseek | 172s | 13 | 204 | yes | — |
| django-11728 | django | deepseek | 348s | 26 | 203 | yes | — |
| django-11740 | django | deepseek | 378s | 41 | 421 | yes | — |
| django-11749 | django | deepseek | 650s | 41 | 133 | yes | — |
| django-11790 | django | deepseek | 238s | 20 | 138 | yes | — |
| django-11815 | django | deepseek | 467s | 29 | 163 | yes | — |
| django-11880 | django | deepseek | 170s | 21 | 203 | yes | — |
| django-11951 | django | deepseek | 287s | 35 | 340 | yes | — |
| django-11964 | django | deepseek | 579s | 23 | 606 | yes | — |
| django-11999 | django | deepseek | 182s | 16 | 61 | yes | — |
| django-12039 | django | deepseek | 291s | 21 | 431 | yes | — |
| django-12050 | django | deepseek | 210s | 18 | 153 | yes | — |
| django-12125 | django | deepseek | 620s | 31 | 943 | yes | — |
| django-12143 | django | deepseek | 152s | 11 | 19 | yes | — |
| django-12155 | django | deepseek | 164s | 13 | 76 | yes | — |
| django-12193 | django | deepseek | 409s | 35 | 260 | yes | — |
| django-12262 | django | deepseek | 303s | 32 | 259 | yes | — |
| django-12276 | django | deepseek | 174s | 14 | 294 | yes | — |
| django-12304 | django | deepseek | 512s | 30 | 525 | yes | — |
| django-12308 | django | deepseek | 273s | 20 | 349 | no | — |
| django-12419 | django | deepseek | 212s | 15 | 408 | yes | — |
| django-12708 | django | deepseek | 406s | 31 | 321 | yes | — |
| django-12713 | django | deepseek | 215s | 18 | 145 | yes | — |
| django-12741 | django | deepseek | 145s | 8 | 138 | yes | — |
| django-12858 | django | deepseek | 490s | 35 | 493 | yes | — |
| django-13012 | django | deepseek | 385s | 24 | 308 | yes | — |
| django-13023 | django | deepseek | 200s | 13 | 252 | yes | — |
| django-13028 | django | deepseek | 498s | 26 | 216 | yes | — |
| django-13089 | django | deepseek | 310s | 21 | 262 | yes | — |
| django-13109 | django | deepseek | 248s | 19 | 127 | yes | — |
The continuation of the section above: multi-agent architectures and deterministic issue analysis, each tried against the instances the main run missed. Same rules -- experiments, not submissions; nothing here combines with the 387; crashes stay in the record as crashes. Rows marked “—” were generated but never graded because the run was cancelled first.
A second agent writes a reproduction test from the issue, in parallel and in isolation; the worker must pass it. Every worker ran the tests heavily and satisfied them — and conversions stayed under re-run variance: the tester, reading the same issue with the same model, encodes the same misreading, and its wrong tests ANCHOR the worker. A mid-run infrastructure failure (a shared writable runtime volume poisoned by an agent shell — since made read-only) crashed 24 rows; they are recorded as crashes. The first 17 clean rows ran a flash tester, the remainder a Sonnet tester after the rebuild.
| instance | repo | config | wall | turns | lines read | resolved | cost |
|---|---|---|---|---|---|---|---|
| pylint-4604 | pylint | flash tester | 195s | 12 | 88 | no | — |
| sphinx-9711 | sphinx | flash tester | 120s | 16 | 92 | no | — |
| pylint-4661 | pylint | flash tester | 335s | 33 | 217 | no | — |
| django-14140 | django | flash tester | 463s | 30 | 344 | no | — |
| django-11239 | django | flash tester | 498s | 31 | 444 | no | — |
| django-10999 | django | flash tester | 548s | 21 | 44 | no | — |
| django-11820 | django | flash tester | 548s | 23 | 176 | no | — |
| django-11532 | django | flash tester | 590s | 31 | 99 | no | — |
| sphinx-7462 | sphinx | flash tester | 260s | 14 | 55 | no | — |
| django-14315 | django | flash tester | 630s | 29 | 336 | yes | — |
| django-14559 | django | flash tester | 742s | — | — | crash | — |
| django-14170 | django | flash tester | 816s | 32 | 461 | no | — |
| sympy-16886 | sympy | flash tester | 862s | — | — | crash | — |
| django-15127 | django | flash tester | 453s | 27 | 354 | no | — |
| django-12774 | django | flash tester | 388s | 31 | 360 | yes | — |
| astropy-7606 | astropy | flash tester | 988s | — | — | crash | — |
| django-11477 | django | flash tester | 4s | — | — | crash | — |
| astropy-14598 | astropy | flash tester | 682s | 36 | 603 | no | — |
| django-14376 | django | flash tester | 5s | — | — | crash | — |
| sphinx-9602 | sphinx | flash tester | 33s | — | — | crash | — |
| sympy-18199 | sympy | flash tester | 33s | — | — | crash | — |
| django-12325 | django | flash tester | 452s | 36 | 300 | no | — |
| pylint-8898 | pylint | flash tester | 18s | — | — | crash | — |
| django-15916 | django | flash tester | 18s | — | — | crash | — |
| sympy-20916 | sympy | flash tester | 21s | — | — | crash | — |
| django-11848 | django | flash tester | 208s | 12 | 115 | no | — |
| django-14155 | django | flash tester | 30s | — | — | crash | — |
| django-14034 | django | flash tester | 40s | — | — | crash | — |
| astropy-14182 | astropy | flash tester | 20s | — | — | crash | — |
| django-12406 | django | flash tester | 21s | — | — | crash | — |
| django-15987 | django | flash tester | 11s | — | — | crash | — |
| sphinx-7748 | sphinx | flash tester | 3s | — | — | crash | — |
| django-13195 | django | flash tester | 295s | 21 | 239 | no | — |
| sphinx-10466 | sphinx | flash tester | 21s | — | — | crash | — |
| django-13297 | django | flash tester | 22s | — | — | crash | — |
| django-11141 | django | flash tester | 20s | — | — | crash | — |
| django-14534 | django | flash tester | 22s | — | — | crash | — |
| sphinx-8056 | sphinx | flash tester | 3s | — | — | crash | — |
| sphinx-7590 | sphinx | flash tester | 6s | — | — | crash | — |
| sphinx-9461 | sphinx | flash tester | 6s | — | — | crash | — |
| sympy-21930 | sympy | flash tester | 5s | — | — | crash | — |
| astropy-7606 | astropy | sonnet tester | 1s | — | — | crash | — |
| django-14559 | django | sonnet tester | 3s | — | — | crash | — |
| django-14011 | django | sonnet tester | 3s | — | — | crash | — |
| django-13794 | django | sonnet tester | 3s | — | — | crash | — |
| django-11149 | django | sonnet tester | 3s | — | — | crash | — |
| django-14376 | django | sonnet tester | 4s | — | — | crash | — |
| django-13513 | django | sonnet tester | 4s | — | — | crash | — |
| xarray-3993 | xarray | sonnet tester | 4s | — | — | crash | — |
| django-11477 | django | sonnet tester | 6s | — | — | crash | — |
| sympy-18763 | sympy | sonnet tester | 6s | — | — | crash | — |
| sphinx-9602 | sphinx | sonnet tester | 5s | — | — | crash | — |
| django-16667 | django | sonnet tester | 7s | — | — | crash | — |
| sympy-16886 | sympy | sonnet tester | 8s | — | — | crash | — |
| django-15916 | django | sonnet tester | 5s | — | — | crash | — |
| sympy-18199 | sympy | sonnet tester | 5s | — | — | crash | — |
| pylint-8898 | pylint | sonnet tester | 6s | — | — | crash | — |
| sympy-20916 | sympy | sonnet tester | 7s | — | — | crash | — |
| django-14034 | django | sonnet tester | 7s | — | — | crash | — |
| django-14155 | django | sonnet tester | 7s | — | — | crash | — |
| astropy-14182 | astropy | sonnet tester | 5s | — | — | crash | — |
| django-12406 | django | sonnet tester | 7s | — | — | crash | — |
| sphinx-7748 | sphinx | sonnet tester | 6s | — | — | crash | — |
| django-15987 | django | sonnet tester | 7s | — | — | crash | — |
| sphinx-10466 | sphinx | sonnet tester | 6s | — | — | crash | — |
| django-13297 | django | sonnet tester | 7s | — | — | crash | — |
| django-14534 | django | sonnet tester | 7s | — | — | crash | — |
| django-11141 | django | sonnet tester | 12s | — | — | crash | — |
| sympy-16886 | sympy | sonnet tester | 1s | — | — | crash | — |
| django-16667 | django | sonnet tester | 3s | — | — | crash | — |
| django-14011 | django | sonnet tester | 3s | — | — | crash | — |
| django-14376 | django | sonnet tester | 3s | — | — | crash | — |
| django-13794 | django | sonnet tester | 4s | — | — | crash | — |
| django-14559 | django | sonnet tester | 4s | — | — | crash | — |
| django-11149 | django | sonnet tester | 4s | — | — | crash | — |
| astropy-7606 | astropy | sonnet tester | 5s | — | — | crash | — |
| xarray-3993 | xarray | sonnet tester | 5s | — | — | crash | — |
| sympy-18763 | sympy | sonnet tester | 5s | — | — | crash | — |
| django-11477 | django | sonnet tester | 6s | — | — | crash | — |
| sympy-18199 | sympy | sonnet tester | 5s | — | — | crash | — |
| sphinx-9602 | sphinx | sonnet tester | 8s | — | — | crash | — |
| pylint-8898 | pylint | sonnet tester | 6s | — | — | crash | — |
| django-13513 | django | sonnet tester | 11s | — | — | crash | — |
| django-15916 | django | sonnet tester | 7s | — | — | crash | — |
| sympy-20916 | sympy | sonnet tester | 7s | — | — | crash | — |
| django-14034 | django | sonnet tester | 7s | — | — | crash | — |
| django-14155 | django | sonnet tester | 7s | — | — | crash | — |
| astropy-14182 | astropy | sonnet tester | 6s | — | — | crash | — |
| sphinx-7748 | sphinx | sonnet tester | 5s | — | — | crash | — |
| django-12406 | django | sonnet tester | 7s | — | — | crash | — |
| django-15987 | django | sonnet tester | 7s | — | — | crash | — |
| sphinx-10466 | sphinx | sonnet tester | 5s | — | — | crash | — |
| django-13297 | django | sonnet tester | 6s | — | — | crash | — |
| django-14534 | django | sonnet tester | 18s | — | — | crash | — |
| sphinx-9461 | sphinx | sonnet tester | 42s | — | — | crash | — |
| django-11141 | django | sonnet tester | 44s | — | — | crash | — |
| sympy-15875 | sympy | sonnet tester | 43s | — | — | crash | — |
| django-16454 | django | sonnet tester | 42s | — | — | crash | — |
| sympy-21930 | sympy | sonnet tester | 42s | — | — | crash | — |
| astropy-13033 | astropy | sonnet tester | 43s | — | — | crash | — |
| sphinx-7590 | sphinx | sonnet tester | 43s | — | — | crash | — |
| sphinx-8056 | sphinx | sonnet tester | 43s | — | — | crash | — |
| django-12273 | django | sonnet tester | 42s | — | — | crash | — |
| django-15695 | django | sonnet tester | 40s | — | — | crash | — |
| xarray-4687 | xarray | sonnet tester | 28s | — | — | crash | — |
| xarray-6938 | xarray | sonnet tester | 104s | — | — | crash | — |
| django-10973 | django | sonnet tester | 92s | — | — | crash | — |
| xarray-6992 | xarray | sonnet tester | 66s | — | — | crash | — |
| django-14792 | django | sonnet tester | 70s | — | — | crash | — |
| django-16502 | django | sonnet tester | 70s | — | — | crash | — |
| django-15252 | django | sonnet tester | 70s | — | — | crash | — |
| django-13344 | django | sonnet tester | 70s | — | — | crash | — |
| django-13212 | django | sonnet tester | 70s | — | — | crash | — |
| django-15037 | django | sonnet tester | 70s | — | — | crash | — |
| sympy-22080 | sympy | sonnet tester | 42s | — | — | crash | — |
| django-16631 | django | sonnet tester | 70s | — | — | crash | — |
| pytest-10356 | pytest | sonnet tester | 70s | — | — | crash | — |
| django-11477 | django | sonnet tester | 294s | 19 | 90 | no | — |
| django-14376 | django | sonnet tester | 346s | 24 | 246 | no | — |
| xarray-3993 | xarray | sonnet tester | 420s | 37 | 166 | no | — |
| django-14559 | django | sonnet tester | 440s | — | — | crash | — |
| django-16667 | django | sonnet tester | 475s | 22 | 212 | no | — |
| astropy-7606 | astropy | sonnet tester | 530s | 19 | 83 | yes | — |
| django-13513 | django | sonnet tester | 556s | 32 | 64 | — | — |
| django-11149 | django | sonnet tester | 581s | 43 | 540 | yes | — |
| django-13794 | django | sonnet tester | 703s | 42 | 395 | no | — |
| sympy-18763 | sympy | sonnet tester | 719s | 18 | 25 | no | — |
| sympy-16886 | sympy | sonnet tester | 729s | 17 | 36 | yes | — |
| django-15916 | django | sonnet tester | 396s | 27 | 319 | no | — |
| django-14155 | django | sonnet tester | 418s | 23 | 136 | no | — |
| sphinx-9602 | sphinx | sonnet tester | 750s | 59 | 628 | no | — |
| django-14034 | django | sonnet tester | 637s | 49 | 478 | no | — |
| django-14011 | django | sonnet tester | 1,225s | 52 | 538 | no | — |
| django-12406 | django | sonnet tester | 683s | 38 | 565 | no | — |
| pylint-8898 | pylint | sonnet tester | 832s | 47 | 781 | no | — |
| django-15987 | django | sonnet tester | 566s | — | — | crash | — |
| sympy-18199 | sympy | sonnet tester | 956s | 36 | 170 | no | — |
| sphinx-10466 | sphinx | sonnet tester | 402s | 52 | 296 | yes | — |
| django-13297 | django | sonnet tester | 683s | 35 | 318 | no | — |
| sympy-20916 | sympy | sonnet tester | 1,095s | 40 | 352 | no | — |
| astropy-14182 | astropy | sonnet tester | 1,094s | — | — | crash | — |
| sphinx-8056 | sphinx | sonnet tester | 683s | 43 | 958 | yes | — |
| django-14534 | django | sonnet tester | 964s | 53 | 444 | no | — |
| sphinx-9461 | sphinx | sonnet tester | 719s | 48 | 798 | no | — |
| sphinx-7590 | sphinx | sonnet tester | 759s | 53 | 589 | no | — |
| django-12273 | django | sonnet tester | 629s | 34 | 236 | no | — |
| django-16454 | django | sonnet tester | 620s | 32 | 340 | no | — |
| django-11141 | django | sonnet tester | 1,109s | — | — | crash | — |
| django-10973 | django | sonnet tester | 313s | 21 | 248 | no | — |
| django-15695 | django | sonnet tester | 644s | 41 | 575 | no | — |
| sympy-21930 | sympy | sonnet tester | 1,126s | 34 | 135 | no | — |
| django-13212 | django | sonnet tester | 552s | 27 | 645 | no | — |
| pytest-10356 | pytest | sonnet tester | 770s | 59 | 331 | no | — |
| xarray-6992 | xarray | sonnet tester | 884s | — | — | crash | — |
| sympy-15875 | sympy | sonnet tester | 1,657s | 83 | 255 | no | — |
| xarray-4687 | xarray | sonnet tester | 936s | 75 | 587 | yes | — |
| astropy-13033 | astropy | sonnet tester | 1,717s | — | — | crash | — |
| django-16631 | django | sonnet tester | 612s | 30 | 344 | no | — |
After the worker finishes, a second model reads only the issue and the diff and rules AGREE or REVISE; a REVISE sends one critique back. Measured: the flash verifier revised 1 run in 32; Sonnet, on different weights, about 1 in 8 — both effectively agree with any plausible-looking diff, and outcomes tracked plain re-run variance. Rows through 33 ran the flash verifier, the rest Sonnet.
| instance | repo | config | wall | turns | lines read | resolved | cost |
|---|---|---|---|---|---|---|---|
| django-14559 | django | flash verifier | 196s | 9 | 59 | yes | — |
| django-11239 | django | flash verifier | 200s | 11 | 138 | no | — |
| django-14315 | django | flash verifier | 283s | 19 | 333 | no | — |
| astropy-7606 | astropy | flash verifier | 309s | 14 | 80 | yes | — |
| django-10999 | django | flash verifier | 330s | 17 | 91 | no | — |
| pylint-4604 | pylint | flash verifier | 344s | 33 | 202 | no | — |
| sphinx-9711 | sphinx | flash verifier | 153s | 18 | 160 | no | — |
| django-14140 | django | flash verifier | 365s | 24 | 46 | no | — |
| pylint-4661 | pylint | flash verifier | 371s | 29 | 446 | no | — |
| django-14170 | django | flash verifier | 416s | 22 | 476 | no | — |
| django-11820 | django | flash verifier | 430s | 21 | 262 | no | — |
| sphinx-7462 | sphinx | flash verifier | 201s | 20 | 90 | no | — |
| django-15127 | django | flash verifier | 202s | 12 | 246 | no | — |
| sympy-16886 | sympy | flash verifier | 530s | 10 | 57 | yes | — |
| django-13513 | django | flash verifier | 221s | 25 | 172 | — | — |
| xarray-3993 | xarray | flash verifier | 264s | 23 | 228 | yes | — |
| django-11532 | django | flash verifier | 608s | 23 | 269 | no | — |
| django-13195 | django | flash verifier | 222s | 20 | 130 | yes | — |
| django-12325 | django | flash verifier | 337s | 20 | 155 | no | — |
| django-12774 | django | flash verifier | 372s | 30 | 328 | no | — |
| django-16667 | django | flash verifier | 200s | 14 | 49 | no | — |
| django-14376 | django | flash verifier | 229s | 19 | 310 | no | — |
| django-11477 | django | flash verifier | 267s | 20 | 79 | no | — |
| django-13794 | django | flash verifier | 453s | 32 | 168 | no | — |
| sphinx-9602 | sphinx | flash verifier | 309s | 33 | 691 | no | — |
| django-11848 | django | flash verifier | 604s | 29 | 202 | no | — |
| astropy-14598 | astropy | flash verifier | 1,041s | 47 | 989 | no | — |
| sympy-18199 | sympy | flash verifier | 643s | 18 | 184 | no | — |
| django-15916 | django | flash verifier | 644s | 36 | 467 | yes | — |
| django-11149 | django | flash verifier | 935s | 51 | 1,094 | yes | — |
| django-14155 | django | flash verifier | 516s | 26 | 341 | no | — |
| django-14034 | django | flash verifier | 534s | 21 | 488 | no | — |
| django-13513 | django | flash verifier | 162s | 16 | 87 | — | — |
| sphinx-10466 | sphinx | sonnet verifier | 300s | 34 | 546 | yes | — |
| django-11141 | django | sonnet verifier | 179s | 13 | 169 | no | — |
| django-14534 | django | sonnet verifier | 394s | 30 | 167 | no | — |
| django-15987 | django | sonnet verifier | 428s | 26 | 250 | no | — |
| astropy-14182 | astropy | sonnet verifier | 432s | 29 | 362 | no | — |
| django-13297 | django | sonnet verifier | 513s | 28 | 473 | — | — |
| django-12406 | django | sonnet verifier | 633s | 25 | 485 | — | — |
| django-14011 | django | sonnet verifier | 678s | 28 | 407 | — | — |
| sphinx-8056 | sphinx | sonnet verifier | 495s | 56 | 470 | — | — |
| sympy-18763 | sympy | sonnet verifier | 880s | 37 | 341 | — | — |
No second agent: the issue’s own evidence (traceback frames, quoted code, named symbols) is resolved against the code index mechanically and handed to the worker before it starts, ranked by convergence. It fired on roughly half the instances and resolved real sites — and conversions stayed at re-run variance, because most misses fail AFTER finding the right place: the dominant failure mode is fix precision, not localization. An offline check against the gold patches had already shown the instances that genuinely mislocate are exactly the ones whose issues carry no resolvable evidence at all.
| instance | repo | config | wall | turns | lines read | resolved | cost |
|---|---|---|---|---|---|---|---|
| pylint-4604 | pylint | symptom map | 152s | 26 | 95 | no | — |
| django-14559 | django | symptom map | 153s | 11 | 312 | no | — |
| django-11239 | django | symptom map | 179s | 17 | 217 | yes | — |
| django-14315 | django | symptom map | 212s | 15 | 328 | yes | — |
| django-14140 | django | symptom map | 216s | 16 | 145 | no | — |
| pylint-4661 | pylint | symptom map | 241s | 30 | 225 | no | — |
| sphinx-9711 | sphinx | symptom map | 111s | 13 | 86 | no | — |
| django-10999 | django | symptom map | 264s | 13 | 149 | no | — |
| django-14170 | django | symptom map | 268s | 23 | 372 | no | — |
| astropy-7606 | astropy | symptom map | 280s | 12 | 125 | yes | — |
| sphinx-7462 | sphinx | symptom map | 107s | 16 | 132 | no | — |
| django-11532 | django | symptom map | 393s | 33 | 312 | no | — |
| django-11820 | django | symptom map | 400s | 20 | 273 | no | — |
| django-15127 | django | symptom map | 212s | 12 | 244 | no | — |
| xarray-3993 | xarray | symptom map | 257s | 28 | 281 | no | — |
| django-12774 | django | symptom map | 250s | 13 | 64 | yes | — |
| sympy-16886 | sympy | symptom map | 511s | 9 | 36 | yes | — |
| django-12325 | django | symptom map | 263s | 20 | 198 | no | — |
| django-13513 | django | symptom map | 286s | 15 | 89 | — | — |
| django-13195 | django | symptom map | 194s | 14 | 172 | no | — |
| django-16667 | django | symptom map | 176s | 12 | 78 | no | — |
| django-13794 | django | symptom map | 384s | 24 | 304 | yes | — |
| django-11848 | django | symptom map | 290s | 13 | 135 | no | — |
| django-11477 | django | symptom map | 256s | 27 | 317 | no | — |
| sphinx-9602 | sphinx | symptom map | 229s | 27 | 211 | no | — |
| astropy-14598 | astropy | symptom map | 691s | 37 | 530 | no | — |
| django-15916 | django | symptom map | 322s | 22 | 295 | no | — |
| django-14376 | django | symptom map | 415s | 30 | 416 | no | — |
| django-14034 | django | symptom map | 319s | 23 | 555 | no | — |
| django-14155 | django | symptom map | 274s | 20 | 220 | no | — |
| django-11149 | django | symptom map | 746s | 51 | 1,354 | yes | — |
| pylint-8898 | pylint | symptom map | 495s | 52 | 516 | no | — |
| django-15987 | django | symptom map | 205s | 16 | 282 | no | — |
| sympy-18763 | sympy | symptom map | 646s | 18 | 32 | yes | — |
| sympy-18199 | sympy | symptom map | 589s | 21 | 353 | no | — |
| astropy-14182 | astropy | symptom map | 376s | 21 | 380 | no | — |
| django-12406 | django | symptom map | 481s | 32 | 640 | no | — |
| django-14011 | django | symptom map | 980s | 44 | 489 | no | — |
| sphinx-10466 | sphinx | symptom map | 256s | 34 | 577 | yes | — |
| django-14534 | django | symptom map | 327s | 26 | 483 | no | — |
| sphinx-7590 | sphinx | symptom map | 296s | 28 | 369 | no | — |
| django-13297 | django | symptom map | 408s | 14 | 316 | no | — |
| django-11141 | django | symptom map | 409s | 42 | 321 | no | — |
| sphinx-7748 | sphinx | symptom map | 614s | 59 | 1,275 | no | — |
| sympy-20916 | sympy | symptom map | 904s | 23 | 306 | — | — |
| astropy-13033 | astropy | symptom map | 420s | 22 | 245 | — | — |
| django-10973 | django | symptom map | 153s | 11 | 187 | — | — |
| xarray-6938 | xarray | symptom map | 249s | 31 | 334 | — | — |
| sphinx-8056 | sphinx | symptom map | 554s | 53 | 561 | — | — |
| django-16454 | django | symptom map | 325s | 19 | 112 | — | — |
| django-12273 | django | symptom map | 432s | 22 | 137 | — | — |
| sympy-15875 | sympy | symptom map | 611s | 13 | 38 | — | — |
| xarray-6992 | xarray | symptom map | 199s | 18 | 357 | — | — |
21 August 2026
A second full pass over all 500 instances with the current engine (worker + scope cards + symptom map + verifier, all on DeepSeek v4-flash), one attempt per instance, graded identically to August run 1. 391 of 500 resolved (78.2%) at a total generation cost of $37.33. Instances whose first attempt died to infrastructure were re-attempted; the table shows each instance’s final attempt.
| Repository | Instances | Resolved | Rate |
|---|---|---|---|
| flask | 1 | 1 | 100.0% |
| requests | 8 | 8 | 100.0% |
| seaborn | 2 | 2 | 100.0% |
| scikit-learn | 32 | 29 | 90.6% |
| pytest | 19 | 16 | 84.2% |
| sympy | 75 | 61 | 81.3% |
| django | 231 | 181 | 78.4% |
| xarray | 22 | 17 | 77.3% |
| sphinx | 44 | 32 | 72.7% |
| matplotlib | 34 | 25 | 73.5% |
| astropy | 22 | 15 | 68.2% |
| pylint | 10 | 4 | 40.0% |
| Instance | Repo | Result | Turns | Lines read | Wall (s) | Cost ($) |
|---|