← All benchmarks

Bug-fix benchmark · the complete record

Every prompt. Every run. Including the ones we lost.

This is the working behind the summary. Every task's verbatim prompt, the upstream repository and the real commit that fixed the bug, the tests used to grade it, and all 578 runs recorded between 2026-07-26 and 2026-08-14 — failures, superseded runs and abandoned experiments included. Nothing here is filtered.

41%
faster than Claude Code
27%
fewer model calls
22
timed bugs on the summary
578
runs recorded in total

The summary reports one run per arm per bug. This page shows every run behind that choice, so a reader can see whether a headline number came from a lucky pick. Where a bug was run more than once for an arm, the summary uses the most recent run on the current engine — not the best one.

How a task is built

Every bug is a real, already-fixed defect from a real repository. Nothing is synthetic and nothing was written for this benchmark.

1. Pick a real fix. A merged commit that repairs a genuine bug and ships a test proving it. That commit's parent is the buggy state; the checkout every agent receives is the parent, so the bug is live and the fix does not exist yet.

2. Write the issue text from the symptom, not the fix. The prompt describes what a user would observe and which behaviour must change. It never names the function to edit or the line to change — that is the work being measured. Every prompt is reproduced verbatim below so this claim can be checked rather than trusted.

3. Hide the answer. The fix commit is in the repository's future, but the history still contains everything up to the parent — and git history is a search index. So for the duration of every run .git is renamed and hidden in BOTH the worktree and the base clone. An agent that reaches for git log gets "not a git repository". This was added after an agent was caught reading a fix out of the base clone.

4. Grade by the repository's own tests. A fix counts only if the target test that failed on the parent now passes AND no previously-passing test in the surrounding suite breaks. Passing the target test while breaking a neighbour is recorded as a regression, not a solve.

5. Give every arm the same checkout. Same commit, same prompt, same model, fresh worktree each time, run back to back on one machine so provider and machine drift hit all arms equally.

The tasks, with their prompts

The exact text handed to every agent, unedited.

addressable ruby

task id addressable_template_nonstring repo sporkmonger/addressable fix commit c00d58b084 buggy parent bcaf766b0a
files the real fix touched lib/addressable/template.rb
graded on spec/addressable/template_spec.rb regression suite
solved Benzi 23/23 · Claude Code 7/7 · Codex CLI 3/3 · OpenCode 1/2 · DeepSeek Harness 1/1
Addressable::Template#expand crashes with a NoMethodError ("undefined method 'to_str' for an instance of Symbol") when expanding an RFC 6570 URI template with a composite (Hash or Array) variable whose leaf values aren't strings -- for example a Hash key that's a Symbol, or an Array of Integers. Fix template expansion so non-string leaf values in composite variables are stringified correctly instead of crashing, and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

cJSON c

task id cjson_null_deref repo DaveGamble/cJSON fix commit b2890c8d76 buggy parent a3f3d6c784
files the real fix touched cJSON.c
graded on misc_tests regression suite misc_tests
solved Benzi 17/18 · Claude Code 5/5 · OpenCode 2/2 · Codex CLI 1/2 · DeepSeek Harness 1/1
cJSON_SetNumberHelper(object, number) dereferences `object` immediately without checking for NULL first, so calling it with a NULL object pointer crashes instead of failing gracefully. It should check for a NULL object at entry and return NAN in that case, consistent with how other cJSON setter functions handle NULL. Fix it and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

commons-cli java

task id commonscli_npe repo apache/commons-cli fix commit 57c08d0d16 buggy parent 4384414639
files the real fix touched src/main/java/org/apache/commons/cli/CommandLine.java
graded on BugsTest regression suite
solved Benzi 22/22 · Claude Code 7/7 · OpenCode 2/2 · Codex CLI 2/2 · DeepSeek Harness 2/2
commons-cli throws a NullPointerException when an option name of null is passed to CommandLine's option lookup. For example `commandLine.getOptionValue((String) null, "default")` should return the supplied default ("default"), and `commandLine.getOptionValue((String) null, null)` should return null -- but instead a null option name blows up with an NPE while resolving the option. Make a null option name resolve to no option (returning null / the default) rather than throwing, and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

CsvHelper c#

task id csvhelper_nullable repo JoshClose/CsvHelper fix commit 9a7f093463 buggy parent 8599e80f2a
files the real fix touched src/CsvHelper/TypeConversion/TypeConverter.cs
graded on FullyQualifiedName~TypeConverter1Tests regression suite FullyQualifiedName~TypeConverter1Tests
solved Benzi 21/21 · Claude Code 5/5 · Codex CLI 2/2 · OpenCode 1/1 · DeepSeek Harness 1/1
A custom type converter that derives from CsvHelper's generic TypeConverter<T> base, where T is a nullable value type (for example a converter for bool?), throws System.InvalidCastException when it is asked to convert a null value to a string. Writing a null nullable value should serialize normally to an empty string, not blow up. Fix the generic converter so converting a null of a nullable type to string works, and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

dayjs js

task id dayjs_duration_round repo iamkun/dayjs fix commit a9d7d0398d buggy parent 5f3f8786c7
files the real fix touched src/plugin/duration/index.js
graded on test/plugin/duration.test.js regression suite test/plugin/duration.test.js
solved Benzi 18/18 · Claude Code 6/6 · OpenCode 2/2 · Codex CLI 2/2 · DeepSeek Harness 1/1
dayjs's duration formatting produces sub-second values polluted with floating-point noise. For example dayjs.duration(-2812).toISOString() returns '-PT2.8120000000000003S' instead of '-PT2.812S', and dayjs.duration(3121632.27382247).toISOString() returns 'PT52M1.6320000000000001S' instead of 'PT52M1.632S'. The sub-second component of the ISO string is not rounded to millisecond precision. Fix the duration ISO serialization so sub-second values are rounded to at most millisecond precision, and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

fmt c++

task id fmt_4839 repo fmtlib/fmt fix commit 7852fc384c buggy parent d45179c1e7
files the real fix touched include/fmt/format.h
graded on format-test regression suite format-test
solved Benzi 15/18 · Claude Code 6/7 · Codex CLI 2/2 · DeepSeek Harness 1/1
fmt mishandles out-of-range integers formatted with the 'c' presentation type. `fmt::format("{:c}", 256)`, a negative value like `fmt::format("{:c}", -1)`, and other values outside a single character's range are silently mistranslated instead of being rejected. The 'c' type treats character values as unsigned, so the representable range is [0, 255]; anything outside it (negative, or > 255) should raise a clear "character value out of range" error, while in-range values (e.g. 200, 255) must still format to the corresponding character. Please fix it and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

gson java

task id gson_duplicate_key_null repo google/gson fix commit f4d371d29c buggy parent 27d9ba1eee
files the real fix touched gson/src/main/java/com/google/gson/internal/bind/MapTypeAdapterFactory.java
graded on MapTest,MapAsArrayTypeAdapterTest regression suite MapTest,MapAsArrayTypeAdapterTest
solved Benzi 22/22 · Claude Code 5/5 · Codex CLI 2/2 · OpenCode 1/1 · DeepSeek Harness 1/1
Gson's Map deserializer detects duplicate JSON object keys by checking whether Map.put(key, value) returns a non-null previous value -- but put() legitimately returns null when the key's FIRST occurrence had a null value, so {"a":null,"a":1} silently deserializes to {a=1} instead of throwing JsonSyntaxException. Fix the duplicate-key check (at both the plain-object call site and the enableComplexMapKeySerialization array-of-entries call site) so it correctly detects a repeated key regardless of the first value, and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

hashie ruby

task id hashie_deep_merge_dup repo hashie/hashie fix commit 4f014e7fed buggy parent 641bafcb44
files the real fix touched lib/hashie/extensions/deep_merge.rb, lib/hashie/utils.rb
graded on spec/hashie/extensions/deep_merge_spec.rb regression suite spec/hashie/extensions/deep_merge_spec.rb
solved Benzi 8/8 · Claude Code 3/3 · Codex CLI 2/2 · DeepSeek Harness 1/1
Hashie::Extensions::DeepMerge#deep_merge uses a shallow dup before merging, so nested hash values inside the receiver are still shared references with the original -- mutating the 'merged' result silently corrupts the original hash's nested structures too. Fix deep_merge so it produces a true deep copy (nested hashes included) without raising on Ruby values that can't be dup'd (Symbol, true/false, nil, Integer, Rational, Complex, Method, ...), and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

http-parser c

task id httpparser_line_folding repo nodejs/http-parser fix commit 76f0f1690f buggy parent 5d9c382172
files the real fix touched http_parser.c
graded on regression suite
solved Benzi 3/8 · Claude Code 1/4 · DeepSeek Harness 1/1
The HTTP header-value parser mishandles a header whose value is empty on the field's own line and only appears on an RFC-2616 line-folding continuation line (e.g. "Connection:\r\n close\r\n"). When the parser hits the CR of the empty first line, it prematurely resets its internal header state and fires the value callback before it knows a continuation line follows. This means 'interesting' headers (Connection, Content-Length, Transfer-Encoding) whose real value only arrives via a fold are never recognized as such -- e.g. 'Connection: close' split this way fails to set the close flag, so keep-alive stays on when it must be off. Fix the state machine so it looks ahead before deciding a header value is complete, and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

jsoup java

task id jsoup_negative_nth_child repo jhy/jsoup fix commit 62674f903b buggy parent d5acae96d5
files the real fix touched src/main/java/org/jsoup/select/QueryParser.java, CHANGES.md
graded on SelectorTest regression suite
solved Benzi 19/19 · Claude Code 4/4 · Codex CLI 2/3 · OpenCode 1/2 · DeepSeek Harness 1/1
jsoup's CSS selector engine mis-parses an :nth-child argument whose step has a minus sign but no digit, such as ":nth-child(-n+2)". The leading minus is dropped, so the step is treated as +1 rather than -1 and the selector matches the wrong elements. Parsing the document "<p>1</p> <p>2</p> <p>3</p> <p>4</p>", the query "p:nth-child(-n+2)" should select exactly the first two paragraphs (own text "1" and "2"), which is what the equivalent explicit-digit form "p:nth-child(-1n+2)" already returns -- but the digit-less form returns more elements than that. Consequently a combined query like "p:nth-child(n+2):nth-child(-n+2)", which should narrow to just "2", is also wrong. The same argument parsing is shared by :nth-last-child, :nth-of-type and :nth-last-of-type, so those are affected too. All the other argument forms must keep working exactly as they do now: the positive digit-less step "p:nth-child(n+2)" (selecting "2", "3", "4"), explicit-digit steps such as "p:nth-child(2n+2)" (selecting "2", "4") and "p:nth-child(-1n+2)", plain numeric offsets, the "odd" and "even" keywords, and the SelectorParseException raised for genuinely unparseable arguments. Please fix it and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

marked js

task id marked_lexer_linebreaks repo markedjs/marked fix commit a9696e2898 buggy parent 6aacd133e4
files the real fix touched src/Lexer.js, src/Tokenizer.js
graded on test/unit/Lexer-spec.js test/unit/marked-spec.js regression suite test/unit/Lexer-spec.js test/unit/marked-spec.js
solved Benzi 7/7 · Claude Code 4/4 · OpenCode 1/1 · DeepSeek Harness 1/1
Markdown lists are sometimes misclassified as 'loose' (rendering with <p> tags inside <li>) when they shouldn't be, and single trailing newlines inside tokens are silently dropped instead of being retained in the token stream. The Lexer's line-accumulation logic and the Tokenizer's list-parsing logic each track newlines independently and disagree about how many consecutive line breaks should mark a list as loose vs. tight. Trace how the Lexer accumulates raw text between tokens and how the Tokenizer counts line breaks per list item, then fix both so a single newline is preserved but only multiple consecutive line breaks trigger loose-list formatting, and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

money ruby

task id money_divide_by_zero repo RubyMoney/money fix commit f8582eb85c buggy parent b751843df2
files the real fix touched lib/money/money/arithmetic.rb, CHANGELOG.md
graded on spec/money/arithmetic_spec.rb regression suite
solved Benzi 16/20 · Codex CLI 3/7 · Claude Code 4/4 · OpenCode 0/2 · DeepSeek Harness 1/1
Money's division and divmod operations don't guard against a zero divisor, so dividing by zero produces nonsense instead of failing cleanly. Money.new(100, 'USD') / Money.new(0, 'USD') raises nothing at all and yields an infinite amount; Money.new(100, 'USD') / 0 (and / 0.0, / BigDecimal('0')) blows up with an unrelated "ArgumentError: must be initialized with a finite value" thrown from deep inside Money's own constructor; and Money.new(100, 'USD').divmod(Money.new(0, 'USD')) / Money.new(100, 'USD').divmod(0) leak BigDecimal's low-level "divided by 0" error instead of anything meaningful about money. All of these should fail fast with a ZeroDivisionError raised by Money itself: the message must be "divided by Money(0)" when the divisor is a zero-valued Money, and "divided by zero" when the divisor is a plain numeric zero (Integer, Float or BigDecimal). Division and modulo by non-zero values must behave exactly as before. Please fix it and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

mux go

task id mux_hostport_vars repo gorilla/mux fix commit 98cb6bf42e buggy parent 948bec34b5
files the real fix touched regexp.go
graded on TestHostMatcher regression suite
solved Benzi 26/30 · Claude Code 8/8 · Codex CLI 4/8 · OpenCode 3/3 · Aider 2/2 · DeepSeek Harness 1/1
gorilla/mux's Host() route matcher lets you omit the port from a host pattern (e.g. Host("{subdomain}.example.com")), and such a route should still match requests whose Host header includes a port, ignoring the port entirely. Route matching itself already does this correctly. But when a route matches and mux extracts the named variables from the host into the request's variables map, it does not ignore the port the same way -- so for a wildcard-port host pattern, the extracted variables are wrong or missing whenever the incoming request's Host header includes a port. Fix variable extraction so it treats the port the same way route matching does, and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

nats-server go

task id natsserver_oversized_publish_raft repo nats-io/nats-server fix commit dce0b028b5 buggy parent a0f553bbcf
files the real fix touched server/filestore.go, server/jetstream_batching.go, server/stream.go
graded on TestJetStreamRejectLargePublishes|TestJetStreamClusterRejectLargePublishesBeforeProposal regression suite
solved Benzi 4/6 · Claude Code 2/4
JetStream file-storage streams must reject a publish whose message exceeds the file store's max on-disk record size with a 'message too large' error, and the stream must stay healthy afterward. The size check only exists deep in the file store's low-level write path, guarded on a single-server (R1) stream by a later consistency-check gate -- but that gate is explicitly disabled for clustered/replicated (R3) streams. So for a replicated stream, an oversized publish sails past every pre-check, gets proposed to and committed by the Raft group, and only then does each replica discover -- while applying the already-committed entry -- that the file store refuses to write it, leaving the stream unable to accept further publishes. Fix this so an oversized message is rejected BEFORE it ever reaches Raft, on both the single-server and clustered pre-proposal paths, and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

nlohmann-json c++

task id nlohmannjson_diagnostic_offsets repo nlohmann/json fix commit bacdabd176 buggy parent d5647e6a3b
files the real fix touched include/nlohmann/detail/input/lexer.hpp, include/nlohmann/detail/input/json_sax.hpp
graded on test-diagnostic-positions_cpp11 regression suite test-diagnostic-positions_cpp11
solved Benzi 11/11 · Claude Code 5/5 · Codex CLI 2/2 · DeepSeek Harness 1/1
nlohmann::json's diagnostic-positions feature (JSON_DIAGNOSTIC_POSITIONS) lets a parsed value report start_pos()/end_pos() -- byte offsets into the original source. For strings, the start offset is currently derived by subtracting the PARSED string's length from the end offset. This is wrong whenever the string contains escape sequences (\n, \t, \", \\, etc.): the source token is longer than the decoded value, so the computed start lands inside the string instead of at the opening quote. Fix it so start_pos()/end_pos() are correct for any mix of escapes and plain/multi-byte UTF-8 content, across both the DOM and streaming/callback SAX parser paths, and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

quartznet c#

task id quartznet_misfire_reschedule repo quartznet/quartznet fix commit 417a5b20ce buggy parent bac5084d1b
files the real fix touched src/Quartz/Impl/Triggers/AbstractTrigger.cs, src/Quartz/Impl/Triggers/SimpleTriggerImpl.cs, src/Quartz/Impl/Triggers/CronTriggerImpl.cs, src/Quartz/Impl/Triggers/CalendarIntervalTriggerImpl.cs, src/Quartz/Impl/Triggers/DailyTimeIntervalTriggerImpl.cs, src/Quartz/Impl/Triggers/RecurrenceTriggerImpl.cs, src/Quartz/Impl/AdoJobStore/JobStoreSupport.cs, src/Quartz/Simpl/RAMJobStore.cs, src/Quartz/SPI/IOperableTrigger.cs
graded on FullyQualifiedName~RescheduleNextWithExistingCount_PastStartTime_DoesNotFireImmediately regression suite FullyQualifiedName~RescheduleNextWithExistingCount_PastStartTime_DoesNotFireImmediately
solved Benzi 9/9 · Claude Code 4/4 · DeepSeek Harness 1/1 · Codex CLI 1/1
Since v3.17, triggers using the DoNothing/RescheduleNextWith* misfire-handling instructions compute the rescheduled fire time using a time shifted back by the misfire threshold, instead of the actual current time. This means a trigger with a past StartAt and e.g. WithMisfireHandlingInstructionNextWithExistingCount can fire immediately at scheduler startup instead of waiting for its next legitimately scheduled occurrence, because the computed fire time can land inside the misfire-threshold window. Fix this across all trigger implementations and the job stores that call their misfire-handling methods, restoring the 'does not want to be fired now' semantics, while guarding against infinite loops when a calendar excludes all future times, and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

rich python

task id rich_softwrap_style_newline repo Textualize/rich fix commit 39ee57dfe7 buggy parent 05ff970926
files the real fix touched rich/console.py
graded on tests/test_text.py::test_soft_wrap_styled regression suite tests
solved Benzi 6/6 · Claude Code 2/2 · Codex CLI 2/2 · DeepSeek Harness 1/1
When Console.print() is given a style AND soft_wrap=True, the style is wrongly applied to the line break itself, so the background colour bleeds past the end of the printed text instead of being reset at the end of the line. For example, with `console = Console(color_system="standard", width=80, force_terminal=True)` then `console.print("soft wrap is on", style="blue on white", soft_wrap=True)` followed by `console.print("Next line")` emits '\x1b[34;47msoft wrap is on\x1b[0m\x1b[34;47m\n\x1b[0mNext line\n' -- the styling sequence is re-opened around the newline and only reset afterwards. The expected output is '\x1b[34;47msoft wrap is on\x1b[0m\nNext line\n': the style must be closed BEFORE the newline, so soft-wrapped styled output does not paint the newline or leak its background into what follows. Note that the non-soft-wrap path already gets this right; only the soft-wrap path is affected. Fix it and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

scrapy python

task id scrapy_proxy_auth_leak repo scrapy/scrapy fix commit fada8be1db buggy parent b7824db573
files the real fix touched scrapy/core/downloader/handlers/_base_streaming.py, scrapy/core/downloader/handlers/_httpx.py
graded on tests/test_downloader_handler_httpx.py::TestMitmProxy::test_proxy_redirect regression suite tests/test_downloader_handler_httpx.py::TestMitmProxy
solved Benzi 10/14 · Claude Code 4/4 · Codex CLI 2/2 · OpenCode 1/1 · DeepSeek Harness 1/1
BaseStreamingDownloadHandler extracts proxy credentials from the request by popping the Proxy-Authorization header off of request.headers, which mutates the shared Request object in place. When a request is redirected, Scrapy builds the new request from the original's headers -- but the header is already gone by then, so the redirected request goes out through the proxy with no Proxy-Authorization at all and the proxy rejects it. Fix proxy credential extraction so it reads the header without mutating the request, while still making sure Proxy-Authorization is never forwarded to the destination server itself, and keep the existing proxy test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

semver rust

task id semver_prerelease repo dtolnay/semver fix commit 5742fc2f58 buggy parent 5742fc2f58
files the real fix touched src/eval.rs
graded on regression suite
solved Benzi 7/10 · Claude Code 3/3 · Codex CLI 2/2 · DeepSeek Harness 1/1
semver's VersionReq matching wrongly accepts a prerelease of an excluded upper bound. For example the requirement `>=1.2.0-0, <1.2` should NOT match the version `1.2.0-alpha.1`: the `<1.2` bound excludes every 1.2.0 release, and that must include 1.2.0's prereleases -- but it is currently reported as matching. More generally, a `<I.J` bound must not match `I.J.0` prereleases. Fix the version-requirement matching so a `<` (and `<=`) bound correctly excludes prereleases of its boundary, and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

sqlglot python

task id sqlglot_7920 repo tobymao/sqlglot fix commit c368c5eb2d buggy parent c4020942
files the real fix touched sqlglot/generators/duckdb.py
graded on tests/dialects/test_bigquery.py regression suite tests/dialects
solved Benzi 14/17 · Codex CLI 2/5 · Claude Code 4/4 · DeepSeek Harness 1/1
Transpiling BigQuery's ARRAY_CONCAT_AGG to DuckDB produces wrong SQL when the aggregate has an ORDER BY or a LIMIT inside it. For example `SELECT ARRAY_CONCAT_AGG(arr ORDER BY y) FROM ...` and `SELECT ARRAY_CONCAT_AGG(arr LIMIT 2) FROM ...` do not round-trip correctly to the duckdb dialect. LIMIT inside ARRAY_CONCAT_AGG has no DuckDB equivalent and should be reported as unsupported rather than silently mistranslated. Please fix the transpilation and make sure the existing test suite still passes. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

sqlparser-rs rust

task id sqlparser_exponential_backtrack repo apache/datafusion-sqlparser-rs fix commit fd5c1633a8 buggy parent b3760221fa
files the real fix touched src/parser/mod.rs
graded on regression suite
solved Benzi 7/8 · Claude Code 4/5 · DeepSeek Harness 1/1
Parsing deeply-nested parenthesized table factors, e.g. SELECT 1 FROM ((((... with many levels of nesting, causes catastrophic (exponential) parse time instead of failing or succeeding quickly. In parse_table_factor, a speculative attempt to parse a derived table fails and backtracks, but the nested-join fallback path then recurses back into parse_table_factor over the same remaining paren chain and retries the identical speculative parse -- doubling work at every nesting level. Find and fix the root cause so parsing such a chain completes in bounded time, and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

ts-pattern ts

task id tspattern_ismatching repo gvergnaud/ts-pattern fix commit 470e7b9096 buggy parent 1520b57b07
files the real fix touched src/is-matching.ts
graded on tests/is-matching.test.ts regression suite tests/is-matching.test.ts
solved Benzi 8/8 · Claude Code 2/3 · OpenCode 0/1 · DeepSeek Harness 1/1
ts-pattern's isMatching(pattern, value), the two-argument form of its type-guard helper, fails to type-check for almost any pattern that isn't a plain object pattern -- array patterns (P.array(P.number)), primitive patterns (P.number, P.boolean), literal patterns (1, 'oops'), and union/intersection patterns all throw a compile error, even though the equivalent curried form isMatching(pattern)(value) works fine. Fix the type signature so non-object patterns type-check correctly in the two-argument form too, while still preserving the original object-pattern behavior (matching against objects with extra unknown properties), and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

yaml-cpp c++

task id yamlcpp_base64 repo jbeder/yaml-cpp fix commit 80ea0028c7 buggy parent 2e6383d272
files the real fix touched src/binary.cpp
graded on yaml-cpp-tests regression suite yaml-cpp-tests
solved Benzi 16/18 · Claude Code 7/7 · Codex CLI 2/2 · OpenCode 1/1 · DeepSeek Harness 1/1
yaml-cpp's base64 decoder (YAML::DecodeBase64) does not reject malformed input. A base64 string whose length is not a valid multiple (i.e. truncated -- leftover characters that don't complete a 4-character group) should be treated as invalid and decode to an EMPTY result, but instead it silently decodes the partial input and returns garbage bytes. Make DecodeBase64 return an empty vector when the input is truncated / has an invalid number of base64 characters, and keep valid base64 decoding correct. Keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

zod ts

task id zod_pipe_payload_flag repo colinhacks/zod fix commit c2be4f8190 buggy parent 1cab69383f
files the real fix touched packages/zod/src/v4/core/schemas.ts, packages/zod/src/v4/classic/schemas.ts
graded on packages/zod/src/v4/classic/tests/catch.test.ts packages/zod/src/v4/classic/tests/preprocess.test.ts regression suite packages/zod/src/v4/classic/tests/catch.test.ts packages/zod/src/v4/classic/tests/preprocess.test.ts
solved Benzi 10/13 · Claude Code 2/5 · Codex CLI 2/2 · OpenCode 0/1 · DeepSeek Harness 1/1
Zod's internal ParsePayload carries a caught/fallback flag so an outer .optional() knows to override a recovered value with undefined when the original input was undefined. This flag isn't propagated correctly through pipe boundaries (.pipe(), used internally by .transform() and .preprocess()): the pipe result handler builds a fresh payload for the right side of a pipe without carrying the flag forward, so a chain like z.string().catch('X').pipe(z.string()).optional().parse(undefined) wrongly returns 'X' instead of undefined. Separately, z.object({ a: z.preprocess(fn, T) }).parse({}) -- a preprocess function meant to fill in a value for a missing key -- throws an 'expected nonoptional' error instead of running fn with undefined. Fix both, and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

dry-schema rubynot on the summary

task id dryschema_maybe_hash_array repo dry-rb/dry-schema fix commit 31e76cb796 buggy parent e33c59236b
files the real fix touched lib/dry/schema/key_validator.rb
graded on spec/integration/schema/unexpected_keys_spec.rb regression suite spec/integration/schema/unexpected_keys_spec.rb
solved Claude Code 0/1 · Benzi 0/1
In dry-schema, when config.validate_keys = true is enabled, a schema containing required(:locations).array(:hash) { required(:feedback_location).maybe(:hash) { ... } } incorrectly raises an "is not allowed" unexpected-key error when feedback_location is nil inside an array element. Since the field is declared maybe(:hash), nil should be accepted without error. Fix the key validator so nil values for optional nested hashes inside arrays aren't flagged as unexpected keys, and keep the existing test suite passing. The repository is at the current working directory. Find and fix the bug. Do not modify any test files.

Every run recorded

All 578 runs, grouped by bug, oldest first. Times include the index build for Benzi. A blank cost means the provider returned no usage for that run.

complete run log · 2026-07-26 to 2026-08-14
whenbugharnessmodelwallcallsfixedcost
addressable_template_nonstring
2026-08-07 13:21addressable_template_nonstringBenzideepseek126s13yes$0.039
2026-08-07 14:00addressable_template_nonstringBenzisonnet38s9yes$0.352
2026-08-10 23:47addressable_template_nonstringBenzideepseek70s7yes$0.027
2026-08-11 05:03addressable_template_nonstringBenzisonnet59s9yes$0.421
2026-08-11 07:36addressable_template_nonstringClaude Codesonnet76s11yes$0.316
2026-08-11 11:32addressable_template_nonstringClaude Codesonnet55s12yes$0.346
2026-08-11 12:54addressable_template_nonstringClaude Codesonnet52s11yes$0.309
2026-08-11 12:55addressable_template_nonstringBenzisonnet71s14yes$0.547
2026-08-11 12:56addressable_template_nonstringBenzideepseek69s9yes$0.029
2026-08-11 13:07addressable_template_nonstringClaude Codesonnet46s10yes$0.295
2026-08-11 13:08addressable_template_nonstringBenzisonnet66s13yes$0.527
2026-08-11 13:10addressable_template_nonstringBenzideepseek78s10yes$0.024
2026-08-11 16:29addressable_template_nonstringBenzisonnet57s10yes$0.469
2026-08-11 16:53addressable_template_nonstringClaude Codesonnet47s12yes$0.352
2026-08-11 16:54addressable_template_nonstringBenzisonnet60s13yes$0.520
2026-08-11 17:32addressable_template_nonstringOpenCode—175s42yes$0.066
2026-08-11 17:34addressable_template_nonstringBenzideepseek72s6yes$0.026
2026-08-11 19:04addressable_template_nonstringClaude Codesonnet62s13yes$0.371
2026-08-11 19:05addressable_template_nonstringBenzisonnet49s9yes$0.428
2026-08-11 19:21addressable_template_nonstringClaude Codesonnet56s10yes$0.350
2026-08-11 19:22addressable_template_nonstringBenzisonnet57s11yes$0.484
2026-08-11 19:34addressable_template_nonstringOpenCodesonnet626s71no$2.251
2026-08-12 16:11addressable_template_nonstringBenzisonnet78s20yes$0.734
2026-08-12 16:19addressable_template_nonstringBenzideepseek97s8yes$0.036
2026-08-12 16:30addressable_template_nonstringBenzisonnet46s10yes$0.493
2026-08-12 16:36addressable_template_nonstringBenzisonnet42s8yes$0.441
2026-08-12 17:04addressable_template_nonstringBenzisonnet41s11yes$0.504
2026-08-12 18:06addressable_template_nonstringBenzisonnet68s13yes$0.569
2026-08-12 21:53addressable_template_nonstringBenzideepseek73s7yes$0.027
2026-08-13 10:04addressable_template_nonstringBenzigpt39s6yes$0.719
2026-08-13 10:08addressable_template_nonstringCodex CLIgpt223s27yes$0.347
2026-08-13 10:17addressable_template_nonstringBenzigpt25s5yes$0.139
2026-08-13 10:20addressable_template_nonstringCodex CLIgpt161s18yes$0.286
2026-08-13 19:40addressable_template_nonstringBenzideepseek126s11yes$0.043
2026-08-13 19:43addressable_template_nonstringDeepSeek Harnessdeepseek162s25yes$0.042
2026-08-14 06:47addressable_template_nonstringCodex CLIgpt56s13yes$0.204
chrono_reverse_date_iterators (not on the summary)
2026-08-11 10:30chrono_reverse_date_iteratorsClaude Codesonnet1474s61yes$4.836
2026-08-11 10:41chrono_reverse_date_iteratorsBenzisonnet649s80yes$2.317
2026-08-11 10:58chrono_reverse_date_iteratorsBenzideepseek1007s72yes$0.286
2026-08-11 12:55chrono_reverse_date_iteratorsClaude Codesonnet1222s47yes$4.131
2026-08-11 13:00chrono_reverse_date_iteratorsBenzisonnet291s50yes$1.010
2026-08-12 04:19chrono_reverse_date_iteratorsClaude Codesonnet807s29yes$1.910
cjson_null_deref
2026-08-07 13:05cjson_null_derefBenzideepseek207s44yes$0.085
2026-08-07 14:03cjson_null_derefBenzisonnet56s8yes$0.191
2026-08-10 23:55cjson_null_derefBenzideepseek103s21yes$0.034
2026-08-11 05:05cjson_null_derefBenzisonnet79s12yes$0.307
2026-08-11 07:38cjson_null_derefClaude Codesonnet92s17yes$0.435
2026-08-11 07:42cjson_null_derefClaude Codesonnet87s19yes$0.563
2026-08-11 11:06cjson_null_derefClaude Codesonnet120s27yes$0.757
2026-08-11 13:45cjson_null_derefClaude Codesonnet118s20yes$0.488
2026-08-11 13:47cjson_null_derefBenzisonnet119s18yes$0.349
2026-08-11 13:52cjson_null_derefBenzideepseek312s34yes$0.093
2026-08-11 15:28cjson_null_derefBenzisonnet71s13yes$0.367
2026-08-11 16:35cjson_null_derefBenzisonnet77s13yes$0.339
2026-08-11 18:04cjson_null_derefOpenCodedeepseek179s43yes$0.047
2026-08-11 18:07cjson_null_derefBenzideepseek125s23yes$0.034
2026-08-11 20:45cjson_null_derefClaude Codesonnet75s17yes$0.442
2026-08-11 20:48cjson_null_derefBenzisonnet121s20yes$0.409
2026-08-11 20:52cjson_null_derefOpenCodesonnet216s18yes$0.250
2026-08-12 16:14cjson_null_derefBenzisonnet34s8no$0.198
2026-08-12 16:23cjson_null_derefBenzideepseek106s30yes$0.039
2026-08-12 16:40cjson_null_derefBenzisonnet84s9yes$0.240
2026-08-12 17:09cjson_null_derefBenzisonnet93s14yes$0.305
2026-08-12 18:10cjson_null_derefBenzisonnet90s12yes$0.265
2026-08-12 22:04cjson_null_derefBenzideepseek146s29yes$0.036
2026-08-13 10:26cjson_null_derefBenzigpt23s6yes$0.051
2026-08-13 10:28cjson_null_derefCodex CLIgpt147s20yes$0.262
2026-08-13 19:52cjson_null_derefBenzideepseek121s22yes$0.036
2026-08-13 19:57cjson_null_derefDeepSeek Harnessdeepseek268s41yes$0.073
2026-08-14 06:50cjson_null_derefCodex CLIgpt47s11no$0.154
commonscli_npe
2026-07-26 17:34commonscli_npeClaude Codesonnet154s21yes$0.437
2026-07-26 17:35commonscli_npeBenzideepseek50s15yes$0.047
2026-07-26 17:37commonscli_npeClaude Codesonnet147s32yes$0.703
2026-07-26 17:38commonscli_npeBenzideepseek43s12yes$0.023
2026-07-29 11:06commonscli_npeClaude Codesonnet103s17yes$0.408
2026-07-29 11:08commonscli_npeBenzideepseek89s22yes$0.053
2026-08-07 12:47commonscli_npeBenzideepseek68s12yes$0.034
2026-08-07 13:56commonscli_npeBenzisonnet64s6yes$0.426
2026-08-08 15:36commonscli_npeBenzideepseek156s13yes$0.048
2026-08-10 23:53commonscli_npeBenzideepseek152s17yes$0.059
2026-08-11 05:06commonscli_npeBenzisonnet75s7yes$0.512
2026-08-11 11:34commonscli_npeClaude Codesonnet86s13yes$0.389
2026-08-11 13:22commonscli_npeClaude Codesonnet89s13yes$0.353
2026-08-11 13:23commonscli_npeBenzisonnet82s17yes$0.815
2026-08-11 13:25commonscli_npeBenzideepseek99s13yes$0.046
2026-08-11 16:30commonscli_npeBenzisonnet73s6yes$0.489
2026-08-11 17:02commonscli_npeClaude Codesonnet50s10yes$0.310
2026-08-11 17:03commonscli_npeBenzisonnet68s12yes$0.669
2026-08-11 17:50commonscli_npeOpenCodedeepseek359s56yes$0.103
2026-08-11 17:51commonscli_npeBenzideepseek76s15yes$0.042
2026-08-11 20:10commonscli_npeClaude Codesonnet67s9yes$0.290
2026-08-11 20:11commonscli_npeBenzisonnet63s9yes$0.585
2026-08-11 20:14commonscli_npeOpenCodesonnet137s24yes$0.265
2026-08-12 16:13commonscli_npeBenzisonnet70s10yes$0.184
2026-08-12 16:21commonscli_npeBenzideepseek79s20yes$0.026
2026-08-12 16:38commonscli_npeBenzisonnet54s13yes$0.218
2026-08-12 17:07commonscli_npeBenzisonnet63s12yes$0.201
2026-08-12 18:08commonscli_npeBenzisonnet86s14yes$0.257
2026-08-12 21:55commonscli_npeBenzideepseek87s15yes$0.020
2026-08-13 10:23commonscli_npeBenzigpt70s10yes$0.086
2026-08-13 10:25commonscli_npeCodex CLIgpt95s18yes$0.233
2026-08-13 19:36commonscli_npeDeepSeek Harnessdeepseek138s17yes$0.014
2026-08-13 19:49commonscli_npeBenzideepseek141s22yes$0.030
2026-08-13 19:50commonscli_npeDeepSeek Harnessdeepseek65s14yes$0.014
2026-08-14 06:49commonscli_npeCodex CLIgpt69s12yes$0.170
csvhelper_nullable
2026-07-29 11:19csvhelper_nullableClaude Codesonnet89s11yes$0.341
2026-07-29 11:20csvhelper_nullableBenzideepseek96s16yes$0.026
2026-08-07 12:57csvhelper_nullableBenzideepseek160s14yes$0.039
2026-08-07 14:02csvhelper_nullableBenzisonnet111s15yes$0.304
2026-08-07 17:12csvhelper_nullableBenzisonnet65s8yes$0.188
2026-08-08 10:35csvhelper_nullableBenzideepseek112s15yes$0.026
2026-08-08 10:54csvhelper_nullableBenzisonnet98s16yes$0.330
2026-08-08 15:40csvhelper_nullableBenzideepseek216s19yes$0.054
2026-08-10 14:22csvhelper_nullableBenzideepseek96s16yes$0.030
2026-08-10 23:39csvhelper_nullableBenzideepseek108s13yes$0.026
2026-08-11 05:13csvhelper_nullableBenzisonnet81s10yes$0.196
2026-08-11 11:36csvhelper_nullableClaude Codesonnet150s27yes$0.766
2026-08-11 14:01csvhelper_nullableClaude Codesonnet97s14yes$0.453
2026-08-11 14:03csvhelper_nullableBenzisonnet119s19yes$0.351
2026-08-11 14:06csvhelper_nullableBenzideepseek140s18yes$0.040
2026-08-11 16:37csvhelper_nullableBenzisonnet98s14yes$0.268
2026-08-11 17:08csvhelper_nullableClaude Codesonnet132s22yes$0.647
2026-08-11 18:00csvhelper_nullableOpenCodedeepseek245s41yes$0.046
2026-08-11 18:01csvhelper_nullableBenzideepseek101s16yes$0.031
2026-08-11 20:24csvhelper_nullableClaude Codesonnet105s20yes$0.583
2026-08-11 20:26csvhelper_nullableBenzisonnet128s20yes$0.443
2026-08-12 16:47csvhelper_nullableBenzisonnet70s10yes$0.207
2026-08-12 17:12csvhelper_nullableBenzisonnet61s10yes$0.209
2026-08-12 18:15csvhelper_nullableBenzisonnet109s15yes$0.251
2026-08-12 21:58csvhelper_nullableBenzideepseek171s17yes$0.041
2026-08-13 10:35csvhelper_nullableBenzigpt49s5yes$0.078
2026-08-13 10:37csvhelper_nullableCodex CLIgpt114s15yes$0.219
2026-08-13 20:16csvhelper_nullableBenzideepseek179s18yes$0.038
2026-08-13 20:19csvhelper_nullableDeepSeek Harnessdeepseek207s25yes$0.042
2026-08-14 06:53csvhelper_nullableCodex CLIgpt35s10yes$0.135
dayjs_duration_round
2026-07-29 10:55dayjs_duration_roundClaude Codesonnet85s11yes$0.349
2026-07-29 11:01dayjs_duration_roundClaude Codesonnet98s12yes$0.430
2026-07-29 11:04dayjs_duration_roundBenzideepseek202s33yes$0.117
2026-08-07 13:00dayjs_duration_roundBenzideepseek215s18yes$0.077
2026-08-07 14:06dayjs_duration_roundBenzisonnet170s24yes$1.251
2026-08-07 17:14dayjs_duration_roundBenzisonnet102s10yes$0.722
2026-08-10 23:59dayjs_duration_roundBenzideepseek264s26yes$0.103
2026-08-11 05:16dayjs_duration_roundBenzisonnet158s15yes$0.887
2026-08-11 07:40dayjs_duration_roundClaude Codesonnet91s10yes$0.328
2026-08-11 11:03dayjs_duration_roundClaude Codesonnet149s21yes$0.599
2026-08-11 13:55dayjs_duration_roundClaude Codesonnet155s13yes$0.464
2026-08-11 13:57dayjs_duration_roundBenzisonnet124s17yes$0.989
2026-08-11 14:00dayjs_duration_roundBenzideepseek136s14yes$0.068
2026-08-11 15:30dayjs_duration_roundBenzisonnet110s9yes$0.680
2026-08-11 15:37dayjs_duration_roundBenzideepseek139s13yes$0.067
2026-08-11 16:39dayjs_duration_roundBenzisonnet120s10yes$0.744
2026-08-11 18:24dayjs_duration_roundOpenCodedeepseek664s43yes$0.069
2026-08-11 18:28dayjs_duration_roundBenzideepseek267s22yes$0.096
2026-08-11 21:14dayjs_duration_roundClaude Codesonnet164s17yes$0.550
2026-08-11 21:16dayjs_duration_roundBenzisonnet96s14yes$0.867
2026-08-11 21:20dayjs_duration_roundOpenCodesonnet222s31yes$0.543
2026-08-12 17:18dayjs_duration_roundBenzisonnet128s14yes$0.225
2026-08-12 18:21dayjs_duration_roundBenzisonnet108s11yes$0.210
2026-08-12 22:17dayjs_duration_roundBenzideepseek285s25yes$0.053
2026-08-13 10:46dayjs_duration_roundBenzigpt104s11yes$0.086
2026-08-13 10:49dayjs_duration_roundCodex CLIgpt183s18yes$0.303
2026-08-13 20:32dayjs_duration_roundBenzideepseek247s16yes$0.038
2026-08-13 20:35dayjs_duration_roundDeepSeek Harnessdeepseek153s24yes$0.040
2026-08-14 06:57dayjs_duration_roundCodex CLIgpt90s13yes$0.213
dryschema_maybe_hash_array (not on the summary)
2026-08-11 08:16dryschema_maybe_hash_arrayClaude Codesonnet796s81no$3.239
2026-08-11 08:37dryschema_maybe_hash_arrayBenzi—631s—no—
fmt_4839
2026-07-26 16:58fmt_4839Claude Codesonnet68s18yes$0.396
2026-07-26 17:10fmt_4839Benzideepseek720s133no$0.459
2026-07-26 17:12fmt_4839Claude Codesonnet94s21yes$0.461
2026-07-26 17:30fmt_4839Benzideepseek1035s134no$0.543
2026-07-26 17:38fmt_4839Benzisonnet280s56no$2.252
2026-07-29 11:12fmt_4839Claude Codesonnet248s36yes$0.882
2026-07-29 11:17fmt_4839Benzideepseek301s57yes$0.178
2026-08-07 12:45fmt_4839Benzideepseek495s66yes$0.168
2026-08-07 14:52fmt_4839Benzisonnet953s90yes$2.345
2026-08-07 16:14fmt_4839Benzisonnet317s58yes$1.202
2026-08-11 00:40fmt_4839Benzideepseek350s38yes$0.100
2026-08-11 06:23fmt_4839Benzisonnet455s69yes$1.835
2026-08-11 08:00fmt_4839Claude Codesonnet328s56yes$1.739
2026-08-11 12:03fmt_4839Claude Codesonnet175s44no$1.245
2026-08-11 14:29fmt_4839Claude Codesonnet256s44yes$1.283
2026-08-11 14:34fmt_4839Benzisonnet258s37yes$0.738
2026-08-11 14:47fmt_4839Benzideepseek759s91yes$0.328
2026-08-11 16:47fmt_4839Benzisonnet458s73yes$1.506
2026-08-11 21:24fmt_4839Claude Codesonnet199s37yes$1.125
2026-08-11 21:34fmt_4839Benzisonnet596s112yes$2.724
2026-08-12 08:28fmt_4839Benzisonnet714s103yes$2.540
2026-08-12 17:35fmt_4839Benzisonnet985s153yes$4.753
2026-08-12 21:35fmt_4839Benzisonnet223s30yes$0.543
2026-08-13 10:53fmt_4839Benzigpt218s17yes$0.247
2026-08-13 10:57fmt_4839Codex CLIgpt257s24yes$0.360
2026-08-13 20:43fmt_4839Benzideepseek481s50yes$0.142
2026-08-13 20:49fmt_4839DeepSeek Harnessdeepseek335s54yes$0.071
2026-08-14 06:59fmt_4839Codex CLIgpt112s19yes$0.261
gson_duplicate_key_null
2026-08-07 12:03gson_duplicate_key_nullClaude Codesonnet87s16yes$0.370
2026-08-07 12:05gson_duplicate_key_nullBenzideepseek97s9yes$0.017
2026-08-07 13:59gson_duplicate_key_nullBenzisonnet132s16yes$0.309
2026-08-08 10:49gson_duplicate_key_nullBenzideepseek164s15yes$0.020
2026-08-08 11:01gson_duplicate_key_nullBenzisonnet86s14yes$0.212
2026-08-08 15:43gson_duplicate_key_nullBenzideepseek146s16yes$0.023
2026-08-10 14:21gson_duplicate_key_nullBenzideepseek156s10yes$0.022
2026-08-10 23:50gson_duplicate_key_nullBenzideepseek138s14yes$0.018
2026-08-11 05:09gson_duplicate_key_nullBenzisonnet127s11yes$0.220
2026-08-11 07:39gson_duplicate_key_nullClaude Codesonnet146s19yes$0.472
2026-08-11 11:01gson_duplicate_key_nullClaude Codesonnet116s2yes$0.512
2026-08-11 13:33gson_duplicate_key_nullClaude Codesonnet106s1yes$0.456
2026-08-11 13:35gson_duplicate_key_nullBenzisonnet115s16yes$0.295
2026-08-11 13:43gson_duplicate_key_nullBenzideepseek403s29yes$0.035
2026-08-11 16:33gson_duplicate_key_nullBenzisonnet154s19yes$0.337
2026-08-11 18:09gson_duplicate_key_nullOpenCodedeepseek160s26yes$0.030
2026-08-11 18:12gson_duplicate_key_nullBenzideepseek143s15yes$0.030
2026-08-11 20:54gson_duplicate_key_nullClaude Codesonnet134s18yes$0.446
2026-08-11 20:56gson_duplicate_key_nullBenzisonnet90s9yes$0.169
2026-08-12 17:15gson_duplicate_key_nullBenzisonnet111s13yes$0.245
2026-08-12 18:17gson_duplicate_key_nullBenzisonnet101s6yes$0.137
2026-08-12 22:10gson_duplicate_key_nullBenzideepseek126s10yes$0.022
2026-08-13 10:39gson_duplicate_key_nullBenzigpt98s8yes$0.130
2026-08-13 10:41gson_duplicate_key_nullCodex CLIgpt96s11yes$0.120
2026-08-13 15:24gson_duplicate_key_nullBenzigpt169s14yes$0.155
2026-08-13 17:32gson_duplicate_key_nullBenzisonnet140s15yes$0.285
2026-08-13 17:59gson_duplicate_key_nullBenzisonnet93s12yes$0.228
2026-08-13 18:17gson_duplicate_key_nullBenzisonnet91s9yes$0.186
2026-08-13 20:22gson_duplicate_key_nullBenzideepseek120s11yes$0.015
2026-08-13 20:24gson_duplicate_key_nullDeepSeek Harnessdeepseek99s20yes$0.022
2026-08-14 06:54gson_duplicate_key_nullCodex CLIgpt77s11yes$0.176
hashie_deep_merge_dup
2026-08-07 11:56hashie_deep_merge_dupClaude Codesonnet489s46yes$1.453
2026-08-07 12:00hashie_deep_merge_dupBenzideepseek223s35yes$0.076
2026-08-07 14:08hashie_deep_merge_dupBenzisonnet94s21yes$0.592
2026-08-11 00:04hashie_deep_merge_dupBenzideepseek250s31yes$0.087
2026-08-11 05:11hashie_deep_merge_dupBenzisonnet118s20yes$0.716
2026-08-11 08:10hashie_deep_merge_dupClaude Codesonnet592s54yes$2.368
2026-08-12 01:27hashie_deep_merge_dupClaude Codesonnet200s27yes$0.920
2026-08-12 01:29hashie_deep_merge_dupBenzisonnet98s22yes$0.671
2026-08-12 18:31hashie_deep_merge_dupBenzisonnet84s15yes$0.686
2026-08-13 10:58hashie_deep_merge_dupBenzigpt35s6yes$0.123
2026-08-13 11:12hashie_deep_merge_dupCodex CLIgpt784s77yes$1.553
2026-08-13 20:51hashie_deep_merge_dupBenzideepseek119s17yes$0.051
2026-08-13 20:54hashie_deep_merge_dupDeepSeek Harnessdeepseek168s28yes$0.043
2026-08-14 07:00hashie_deep_merge_dupCodex CLIgpt59s13yes$0.208
httpparser_line_folding
2026-08-07 05:25httpparser_line_foldingClaude Codesonnet1s1no—
2026-08-07 05:50httpparser_line_foldingBenzideepseek1500s118no$0.685
2026-08-07 05:57httpparser_line_foldingClaude Codesonnet432s41no$1.770
2026-08-07 06:10httpparser_line_foldingBenzideepseek792s76yes$0.326
2026-08-07 08:39httpparser_line_foldingClaude Codesonnet608s41no$2.051
2026-08-07 15:04httpparser_line_foldingBenzisonnet186s39no$1.263
2026-08-07 16:17httpparser_line_foldingBenzisonnet185s40no$1.279
2026-08-11 01:13httpparser_line_foldingBenzideepseek410s31yes$0.141
2026-08-11 07:29httpparser_line_foldingBenzisonnet219s39no$1.315
2026-08-11 22:55httpparser_line_foldingClaude Codesonnet856s54yes$3.438
2026-08-11 23:02httpparser_line_foldingBenzisonnet395s36no$1.219
2026-08-14 01:55httpparser_line_foldingBenzideepseek1001s65yes$0.360
2026-08-14 02:05httpparser_line_foldingDeepSeek Harnessdeepseek598s79yes$0.225
jsoup_negative_nth_child
2026-08-11 09:52jsoup_negative_nth_childClaude Codesonnet109s16yes$0.449
2026-08-11 09:53jsoup_negative_nth_childBenzisonnet74s9yes$0.198
2026-08-11 09:58jsoup_negative_nth_childBenzideepseek267s21yes$0.065
2026-08-11 13:27jsoup_negative_nth_childClaude Codesonnet84s15yes$0.380
2026-08-11 13:28jsoup_negative_nth_childBenzisonnet69s8yes$0.206
2026-08-11 13:31jsoup_negative_nth_childBenzideepseek134s17yes$0.027
2026-08-11 16:59jsoup_negative_nth_childClaude Codesonnet103s16yes$0.493
2026-08-11 17:01jsoup_negative_nth_childBenzisonnet88s12yes$0.280
2026-08-11 17:41jsoup_negative_nth_childOpenCode—0s—no—
2026-08-11 17:43jsoup_negative_nth_childBenzideepseek140s14yes$0.027
2026-08-11 18:44jsoup_negative_nth_childOpenCodedeepseek634s127yes$0.227
2026-08-11 19:55jsoup_negative_nth_childClaude Codesonnet80s10yes$0.296
2026-08-11 19:56jsoup_negative_nth_childBenzisonnet74s7yes$0.194
2026-08-12 16:15jsoup_negative_nth_childBenzisonnet75s7yes$0.228
2026-08-12 16:28jsoup_negative_nth_childBenzideepseek244s28yes$0.044
2026-08-12 16:42jsoup_negative_nth_childBenzisonnet82s13yes$0.248
2026-08-12 17:10jsoup_negative_nth_childBenzisonnet62s8yes$0.192
2026-08-12 18:12jsoup_negative_nth_childBenzisonnet106s14yes$0.275
2026-08-12 22:07jsoup_negative_nth_childBenzideepseek185s20yes$0.035
2026-08-13 10:30jsoup_negative_nth_childBenzigpt75s10yes$0.191
2026-08-13 10:30jsoup_negative_nth_childCodex CLIgpt0s—no—
2026-08-13 15:25jsoup_negative_nth_childBenzigpt98s12yes$0.225
2026-08-13 15:31jsoup_negative_nth_childCodex CLIgpt317s30yes$0.542
2026-08-13 17:34jsoup_negative_nth_childBenzisonnet111s13yes$0.281
2026-08-13 18:00jsoup_negative_nth_childBenzisonnet69s8yes$0.221
2026-08-13 18:19jsoup_negative_nth_childBenzisonnet76s10yes$0.225
2026-08-13 20:01jsoup_negative_nth_childBenzideepseek237s16yes$0.025
2026-08-13 20:06jsoup_negative_nth_childDeepSeek Harnessdeepseek273s34yes$0.051
2026-08-14 06:51jsoup_negative_nth_childCodex CLIgpt58s11yes$0.235
marked_lexer_linebreaks
2026-08-07 04:51marked_lexer_linebreaksClaude Codesonnet792s41yes$2.772
2026-08-07 05:13marked_lexer_linebreaksBenzideepseek1336s69yes$0.448
2026-08-07 09:07marked_lexer_linebreaksClaude Codesonnet134s22yes$0.777
2026-08-07 09:47marked_lexer_linebreaksClaude Codesonnet816s37yes$2.748
2026-08-07 15:28marked_lexer_linebreaksBenzisonnet446s79yes$4.557
2026-08-11 01:40marked_lexer_linebreaksBenzideepseek1045s148yes$0.710
2026-08-11 06:41marked_lexer_linebreaksBenzisonnet216s32yes$0.740
2026-08-12 02:49marked_lexer_linebreaksClaude Codesonnet814s47yes$3.589
2026-08-12 03:01marked_lexer_linebreaksBenzisonnet730s103yes$3.165
2026-08-12 03:17marked_lexer_linebreaksOpenCodesonnet919s123yes$3.965
2026-08-12 19:12marked_lexer_linebreaksBenzisonnet279s49yes$1.083
2026-08-13 23:14marked_lexer_linebreaksBenzideepseek538s74yes$0.195
2026-08-13 23:19marked_lexer_linebreaksDeepSeek Harnessdeepseek305s38yes$0.126
money_divide_by_zero
2026-08-11 10:00money_divide_by_zeroClaude Codesonnet74s15yes$0.480
2026-08-11 10:02money_divide_by_zeroBenzisonnet109s23yes$0.631
2026-08-11 10:05money_divide_by_zeroBenzideepseek160s15yes$0.044
2026-08-11 13:16money_divide_by_zeroClaude Codesonnet96s21yes$0.672
2026-08-11 13:18money_divide_by_zeroBenzisonnet120s25yes$0.697
2026-08-11 13:20money_divide_by_zeroBenzideepseek117s14yes$0.042
2026-08-11 15:23money_divide_by_zeroBenzisonnet181s38yes$0.974
2026-08-11 15:26money_divide_by_zeroBenzisonnet155s33yes$0.727
2026-08-11 15:39money_divide_by_zeroBenzideepseek114s15yes$0.041
2026-08-11 17:04money_divide_by_zeroClaude Codesonnet58s16yes$0.475
2026-08-11 17:06money_divide_by_zeroBenzisonnet86s18yes$0.485
2026-08-11 17:53money_divide_by_zeroOpenCodedeepseek93s29no$0.021
2026-08-11 17:55money_divide_by_zeroBenzideepseek157s16yes$0.040
2026-08-11 20:16money_divide_by_zeroClaude Codesonnet140s30yes$0.952
2026-08-11 20:18money_divide_by_zeroBenzisonnet97s19yes$0.545
2026-08-11 20:22money_divide_by_zeroOpenCodesonnet219s25no$0.291
2026-08-12 17:16money_divide_by_zeroBenzisonnet58s16yes$0.574
2026-08-12 18:19money_divide_by_zeroBenzisonnet85s24yes$0.788
2026-08-12 22:12money_divide_by_zeroBenzideepseek126s16yes$0.045
2026-08-13 09:32money_divide_by_zeroBenzi—2s—no—
2026-08-13 09:33money_divide_by_zeroCodex CLIgpt25s—no—
2026-08-13 09:35money_divide_by_zeroBenzi—5s—no—
2026-08-13 09:36money_divide_by_zeroCodex CLIgpt31s—no—
2026-08-13 09:42money_divide_by_zeroBenzi—4s—no—
2026-08-13 09:43money_divide_by_zeroCodex CLIgpt23s—no—
2026-08-13 09:48money_divide_by_zeroBenzi—6s—no—
2026-08-13 09:51money_divide_by_zeroCodex CLIgpt142s—no—
2026-08-13 09:57money_divide_by_zeroBenzigpt81s25yes$2.627
2026-08-13 09:59money_divide_by_zeroCodex CLIgpt110s15yes$0.134
2026-08-13 10:42money_divide_by_zeroBenzigpt27s6yes$0.056
2026-08-13 10:44money_divide_by_zeroCodex CLIgpt157s15yes$0.210
2026-08-13 20:25money_divide_by_zeroBenzideepseek96s13yes$0.036
2026-08-13 20:28money_divide_by_zeroDeepSeek Harnessdeepseek158s20yes$0.052
2026-08-14 06:55money_divide_by_zeroCodex CLIgpt39s9yes$0.174
mux_hostport_vars
2026-08-07 13:02mux_hostport_varsBenzideepseek74s11yes$0.019
2026-08-07 13:56mux_hostport_varsBenzisonnet20s4yes$0.291
2026-08-10 14:17mux_hostport_varsBenzideepseek67s11yes$0.027
2026-08-10 23:37mux_hostport_varsBenzideepseek80s14yes$0.033
2026-08-11 05:02mux_hostport_varsBenzisonnet66s10yes$0.450
2026-08-11 07:35mux_hostport_varsClaude Codesonnet51s6yes$0.223
2026-08-11 11:31mux_hostport_varsClaude Codesonnet54s9yes$0.282
2026-08-11 12:50mux_hostport_varsClaude Codesonnet54s9yes$0.267
2026-08-11 12:51mux_hostport_varsBenzisonnet49s10yes$0.455
2026-08-11 12:53mux_hostport_varsBenzideepseek104s18yes$0.045
2026-08-11 13:04mux_hostport_varsClaude Codesonnet36s8yes$0.273
2026-08-11 13:05mux_hostport_varsBenzisonnet55s11yes$0.493
2026-08-11 13:06mux_hostport_varsBenzideepseek56s12yes$0.020
2026-08-11 16:12mux_hostport_varsAider—17s1yes—
2026-08-11 16:18mux_hostport_varsAider—29s1yes$0.005
2026-08-11 16:19mux_hostport_varsBenzisonnet52s10yes$0.441
2026-08-11 16:22mux_hostport_varsBenzideepseek138s22yes$0.055
2026-08-11 16:28mux_hostport_varsBenzisonnet43s8yes$0.404
2026-08-11 16:52mux_hostport_varsClaude Codesonnet54s13yes$0.521
2026-08-11 16:52mux_hostport_varsBenzisonnet49s10yes$0.447
2026-08-11 17:22mux_hostport_varsOpenCode—131s38yes$0.012
2026-08-11 17:28mux_hostport_varsOpenCode—323s60yes$0.120
2026-08-11 17:29mux_hostport_varsBenzideepseek57s10yes$0.018
2026-08-11 18:57mux_hostport_varsClaude Codesonnet184s12yes$0.548
2026-08-11 18:57mux_hostport_varsBenzisonnet49s9yes$0.417
2026-08-11 19:03mux_hostport_varsOpenCodesonnet346s51yes$1.643
2026-08-11 19:19mux_hostport_varsClaude Codesonnet64s14yes$0.393
2026-08-12 08:17mux_hostport_varsClaude Codesonnet61s8yes$0.248
2026-08-12 16:12mux_hostport_varsBenzisonnet57s10yes$0.520
2026-08-12 16:20mux_hostport_varsBenzideepseek50s8yes$0.027
2026-08-12 16:33mux_hostport_varsBenzisonnet38s10yes$0.267
2026-08-12 16:37mux_hostport_varsBenzisonnet41s11yes$0.299
2026-08-12 17:06mux_hostport_varsBenzisonnet89s20yes$0.785
2026-08-12 18:06mux_hostport_varsBenzisonnet46s5yes$0.390
2026-08-12 21:52mux_hostport_varsBenzideepseek101s11yes$0.036
2026-08-13 09:32mux_hostport_varsBenzi—4s—no—
2026-08-13 09:32mux_hostport_varsCodex CLIgpt23s—no—
2026-08-13 09:35mux_hostport_varsBenzi—6s—no—
2026-08-13 09:35mux_hostport_varsCodex CLIgpt22s—no—
2026-08-13 09:42mux_hostport_varsBenzi—5s—no—
2026-08-13 09:42mux_hostport_varsCodex CLIgpt26s—no—
2026-08-13 09:46mux_hostport_varsBenzi—17s—no—
2026-08-13 09:48mux_hostport_varsCodex CLIgpt140s—no—
2026-08-13 09:53mux_hostport_varsBenzigpt41s11yes$1.473
2026-08-13 09:56mux_hostport_varsCodex CLIgpt136s17yes$0.141
2026-08-13 10:09mux_hostport_varsBenzigpt50s6yes$0.792
2026-08-13 10:11mux_hostport_varsCodex CLIgpt130s12yes$0.113
2026-08-13 10:21mux_hostport_varsBenzigpt28s7yes$0.070
2026-08-13 10:22mux_hostport_varsCodex CLIgpt72s14yes$0.229
2026-08-13 19:45mux_hostport_varsBenzideepseek90s12yes$0.036
2026-08-13 19:46mux_hostport_varsDeepSeek Harnessdeepseek75s13yes$0.023
2026-08-14 06:48mux_hostport_varsCodex CLIgpt33s8yes$0.153
natsserver_oversized_publish_raft
2026-08-07 07:00natsserver_oversized_publish_raftClaude Codesonnet388s41yes$1.749
2026-08-07 08:49natsserver_oversized_publish_raftBenzideepseek2293s70yes$0.229
2026-08-07 11:35natsserver_oversized_publish_raftClaude Codesonnet821s91broke tests$4.454
2026-08-07 15:51natsserver_oversized_publish_raftBenzisonnet971s74yes$2.076
2026-08-11 02:10natsserver_oversized_publish_raftBenzideepseek1568s71yes$0.230
2026-08-11 07:21natsserver_oversized_publish_raftBenzisonnet852s43broke tests$1.345
2026-08-11 08:43natsserver_oversized_publish_raftClaude Codesonnet1347s71yes$2.939
2026-08-12 07:27natsserver_oversized_publish_raftClaude Codesonnet902s101broke tests$4.881
2026-08-12 20:01natsserver_oversized_publish_raftBenzisonnet1456s50yes$1.993
2026-08-13 23:45natsserver_oversized_publish_raftBenzideepseek1338s46broke tests$0.158
nlohmannjson_diagnostic_offsets
2026-08-07 06:42nlohmannjson_diagnostic_offsetsClaude Codesonnet286s41yes$1.639
2026-08-07 06:50nlohmannjson_diagnostic_offsetsBenzideepseek487s57yes$0.174
2026-08-07 11:18nlohmannjson_diagnostic_offsetsClaude Codesonnet406s72yes$2.849
2026-08-07 14:35nlohmannjson_diagnostic_offsetsBenzisonnet754s47yes$1.296
2026-08-07 16:08nlohmannjson_diagnostic_offsetsBenzisonnet711s80yes$2.693
2026-08-11 00:34nlohmannjson_diagnostic_offsetsBenzideepseek655s89yes$0.327
2026-08-11 07:04nlohmannjson_diagnostic_offsetsBenzisonnet684s75yes$2.864
2026-08-11 12:00nlohmannjson_diagnostic_offsetsClaude Codesonnet361s58yes$2.185
2026-08-11 14:57nlohmannjson_diagnostic_offsetsClaude Codesonnet601s76yes$3.649
2026-08-11 15:03nlohmannjson_diagnostic_offsetsBenzisonnet331s51yes$1.534
2026-08-12 01:44nlohmannjson_diagnostic_offsetsClaude Codesonnet504s84yes$3.683
2026-08-12 01:51nlohmannjson_diagnostic_offsetsBenzisonnet386s60yes$1.570
2026-08-12 19:02nlohmannjson_diagnostic_offsetsBenzisonnet307s60yes$1.402
2026-08-12 21:12nlohmannjson_diagnostic_offsetsBenzisonnet208s33yes$1.032
2026-08-13 12:07nlohmannjson_diagnostic_offsetsBenzigpt266s29yes$0.461
2026-08-13 12:13nlohmannjson_diagnostic_offsetsCodex CLIgpt393s50yes$0.713
2026-08-13 22:16nlohmannjson_diagnostic_offsetsBenzideepseek510s52yes$0.165
2026-08-13 22:44nlohmannjson_diagnostic_offsetsDeepSeek Harnessdeepseek1659s124yes$0.440
2026-08-14 07:12nlohmannjson_diagnostic_offsetsCodex CLIgpt180s36yes$0.839
quartznet_misfire_reschedule
2026-08-07 07:52quartznet_misfire_rescheduleClaude Codesonnet289s41yes$1.782
2026-08-07 08:07quartznet_misfire_rescheduleBenzideepseek872s73yes$0.297
2026-08-07 11:47quartznet_misfire_rescheduleClaude Codesonnet712s72yes$4.363
2026-08-07 15:20quartznet_misfire_rescheduleBenzisonnet341s39yes$1.300
2026-08-08 10:46quartznet_misfire_rescheduleBenzideepseek641s50yes$0.174
2026-08-08 10:59quartznet_misfire_rescheduleBenzisonnet296s33yes$1.329
2026-08-11 01:06quartznet_misfire_rescheduleBenzideepseek595s60yes$0.204
2026-08-11 06:16quartznet_misfire_rescheduleBenzisonnet487s50yes$1.693
2026-08-11 08:18quartznet_misfire_rescheduleClaude Codesonnet444s53yes$2.455
2026-08-12 00:48quartznet_misfire_rescheduleClaude Codesonnet560s61yes$3.989
2026-08-12 01:00quartznet_misfire_rescheduleBenzisonnet680s70yes$2.838
2026-08-12 19:08quartznet_misfire_rescheduleBenzisonnet347s29yes$0.949
2026-08-13 22:54quartznet_misfire_rescheduleBenzideepseek581s61yes$0.226
2026-08-13 23:05quartznet_misfire_rescheduleDeepSeek Harnessdeepseek656s90yes$0.310
2026-08-14 07:14quartznet_misfire_rescheduleCodex CLIgpt129s24yes$0.431
rich_softwrap_style_newline
2026-08-11 09:44rich_softwrap_style_newlineClaude Codesonnet361s37yes$1.431
2026-08-11 09:46rich_softwrap_style_newlineBenzisonnet95s17yes$0.256
2026-08-11 09:49rich_softwrap_style_newlineBenzideepseek177s21yes$0.048
2026-08-11 23:36rich_softwrap_style_newlineClaude Codesonnet302s36yes$1.296
2026-08-11 23:38rich_softwrap_style_newlineBenzisonnet97s15yes$0.241
2026-08-12 18:33rich_softwrap_style_newlineBenzisonnet124s20yes$0.326
2026-08-13 11:13rich_softwrap_style_newlineBenzigpt109s19yes$0.252
2026-08-13 11:17rich_softwrap_style_newlineCodex CLIgpt203s30yes$0.409
2026-08-13 21:00rich_softwrap_style_newlineBenzideepseek317s62yes$0.112
2026-08-13 21:03rich_softwrap_style_newlineDeepSeek Harnessdeepseek187s40yes$0.053
2026-08-14 07:01rich_softwrap_style_newlineCodex CLIgpt53s14yes$0.170
scrapy_proxy_auth_leak
2026-08-07 04:29scrapy_proxy_auth_leakClaude Codesonnet588s41yes$1.881
2026-08-07 04:38scrapy_proxy_auth_leakBenzideepseek453s40yes$0.109
2026-08-07 09:04scrapy_proxy_auth_leakClaude Codesonnet375s40yes$1.364
2026-08-07 14:23scrapy_proxy_auth_leakBenzisonnet327s30yes$0.804
2026-08-11 00:23scrapy_proxy_auth_leakBenzideepseek472s41yes$0.096
2026-08-11 06:30scrapy_proxy_auth_leakBenzisonnet324s37yes$0.918
2026-08-11 11:53scrapy_proxy_auth_leakClaude Codesonnet651s41yes$1.540
2026-08-11 23:56scrapy_proxy_auth_leakClaude Codesonnet400s40yes$1.350
2026-08-12 00:01scrapy_proxy_auth_leakBenzisonnet292s24yes$0.567
2026-08-12 00:13scrapy_proxy_auth_leakOpenCodesonnet685s40yes$0.796
2026-08-12 18:48scrapy_proxy_auth_leakBenzisonnet372s31yes$0.791
2026-08-13 11:39scrapy_proxy_auth_leakBenzigpt59s9broke tests$0.148
2026-08-13 11:43scrapy_proxy_auth_leakCodex CLIgpt163s21yes$0.337
2026-08-13 15:33scrapy_proxy_auth_leakBenzigpt73s15broke tests$0.159
2026-08-13 17:45scrapy_proxy_auth_leakBenzisonnet632s47yes$1.193
2026-08-13 18:11scrapy_proxy_auth_leakBenzisonnet595s38yes$1.112
2026-08-13 18:26scrapy_proxy_auth_leakBenzisonnet375s29yes$0.732
2026-08-13 18:58scrapy_proxy_auth_leakBenzigpt320s23broke tests$0.609
2026-08-13 19:11scrapy_proxy_auth_leakBenzigpt5260s13broke tests$0.104
2026-08-13 21:20scrapy_proxy_auth_leakBenzideepseek436s33yes$0.103
2026-08-13 21:32scrapy_proxy_auth_leakDeepSeek Harnessdeepseek689s84yes$0.188
2026-08-14 07:05scrapy_proxy_auth_leakCodex CLIgpt121s16yes$0.267
semver_prerelease
2026-08-07 12:54semver_prereleaseBenzideepseek381s21yes$0.092
2026-08-07 14:17semver_prereleaseBenzisonnet120s19yes$0.530
2026-08-11 00:14semver_prereleaseBenzideepseek279s28yes$0.092
2026-08-11 05:18semver_prereleaseBenzisonnet146s20yes$0.581
2026-08-11 07:54semver_prereleaseClaude Codesonnet740s45yes$2.678
2026-08-11 11:30semver_prereleaseClaude Codesonnet391s23yes$1.428
2026-08-12 00:21semver_prereleaseClaude Codesonnet441s23yes$1.432
2026-08-12 00:24semver_prereleaseBenzisonnet170s25yes$0.689
2026-08-12 18:57semver_prereleaseBenzisonnet139s19yes$0.620
2026-08-13 11:56semver_prereleaseBenzigpt322s44broke tests$0.882
2026-08-13 12:02semver_prereleaseCodex CLIgpt373s38yes$0.798
2026-08-13 19:01semver_prereleaseBenzigpt26s10broke tests$0.241
2026-08-13 19:14semver_prereleaseBenzigpt5282s20no$0.377
2026-08-13 22:02semver_prereleaseBenzideepseek437s19yes$0.100
2026-08-13 22:08semver_prereleaseDeepSeek Harnessdeepseek322s33yes$0.097
2026-08-14 07:09semver_prereleaseCodex CLIgpt60s12yes$0.294
sqlglot_7920
2026-07-29 10:46sqlglot_7920Claude Codesonnet264s33yes$0.887
2026-07-29 10:54sqlglot_7920Benzideepseek452s50yes$0.093
2026-08-07 12:36sqlglot_7920Benzideepseek332s33yes$0.042
2026-08-07 14:15sqlglot_7920Benzisonnet401s21yes$0.390
2026-08-08 14:14sqlglot_7920Benzideepseek399s49yes$0.087
2026-08-10 14:29sqlglot_7920Benzideepseek365s28yes$0.046
2026-08-11 00:10sqlglot_7920Benzideepseek348s49yes$0.086
2026-08-11 06:38sqlglot_7920Benzisonnet470s26yes$0.596
2026-08-11 11:42sqlglot_7920Claude Codesonnet274s39yes$1.118
2026-08-11 14:11sqlglot_7920Claude Codesonnet283s2yes$1.154
2026-08-11 14:18sqlglot_7920Benzisonnet458s20yes$0.400
2026-08-11 14:25sqlglot_7920Benzideepseek375s44yes$0.090
2026-08-11 22:12sqlglot_7920Claude Codesonnet310s39yes$1.232
2026-08-11 22:22sqlglot_7920Benzisonnet563s20yes$0.477
2026-08-12 08:45sqlglot_7920Benzisonnet378s27yes$0.654
2026-08-12 18:41sqlglot_7920Benzisonnet493s23yes$0.616
2026-08-13 09:33sqlglot_7920Benzi—12s—no—
2026-08-13 09:33sqlglot_7920Codex CLIgpt25s—no—
2026-08-13 09:36sqlglot_7920Benzi—23s—no—
2026-08-13 09:37sqlglot_7920Codex CLIgpt30s—no—
2026-08-13 09:43sqlglot_7920Benzi—22s—no—
2026-08-13 09:43sqlglot_7920Codex CLIgpt26s—no—
2026-08-13 11:32sqlglot_7920Benzigpt876s32yes$0.521
2026-08-13 11:38sqlglot_7920Codex CLIgpt348s36yes$0.460
2026-08-13 21:08sqlglot_7920Benzideepseek279s21yes$0.033
2026-08-13 21:12sqlglot_7920DeepSeek Harnessdeepseek265s37yes$0.059
2026-08-14 07:02sqlglot_7920Codex CLIgpt42s15yes$0.217
sqlparser_exponential_backtrack
2026-08-07 06:01sqlparser_exponential_backtrackClaude Codesonnet676s41yes$1.899
2026-08-07 06:23sqlparser_exponential_backtrackClaude Codesonnet744s41no$1.839
2026-08-07 06:37sqlparser_exponential_backtrackBenzideepseek2111s84yes$0.330
2026-08-07 11:11sqlparser_exponential_backtrackClaude Codesonnet1786s83yes$5.396
2026-08-07 15:32sqlparser_exponential_backtrackBenzisonnet199s26yes$0.542
2026-08-11 02:22sqlparser_exponential_backtrackBenzideepseek663s40yes$0.152
2026-08-11 06:07sqlparser_exponential_backtrackBenzisonnet315s32yes$0.664
2026-08-11 09:06sqlparser_exponential_backtrackClaude Codesonnet1271s54yes$3.687
2026-08-12 05:59sqlparser_exponential_backtrackClaude Codesonnet1076s61yes$3.219
2026-08-12 08:30sqlparser_exponential_backtrackBenzisonnet9s—no—
2026-08-12 08:38sqlparser_exponential_backtrackBenzisonnet375s35yes$0.824
2026-08-12 21:19sqlparser_exponential_backtrackBenzisonnet393s33yes$0.720
2026-08-14 01:09sqlparser_exponential_backtrackBenzideepseek1485s54yes$0.135
2026-08-14 01:38sqlparser_exponential_backtrackDeepSeek Harnessdeepseek1731s90yes$0.326
tspattern_ismatching
2026-08-07 13:19tspattern_ismatchingBenzideepseek794s60yes$0.263
2026-08-07 15:14tspattern_ismatchingBenzisonnet595s94yes$4.105
2026-08-07 17:06tspattern_ismatchingBenzisonnet183s29yes$0.939
2026-08-11 01:23tspattern_ismatchingBenzideepseek561s85yes$0.261
2026-08-11 05:22tspattern_ismatchingBenzisonnet220s35yes$1.064
2026-08-11 07:48tspattern_ismatchingClaude Codesonnet594s42no$2.061
2026-08-11 11:23tspattern_ismatchingClaude Codesonnet1041s73yes$4.144
2026-08-12 03:34tspattern_ismatchingClaude Codesonnet1060s84yes$4.328
2026-08-12 03:42tspattern_ismatchingBenzisonnet469s74yes$2.488
2026-08-12 04:06tspattern_ismatchingOpenCodesonnet1392s107no$3.700
2026-08-12 21:30tspattern_ismatchingBenzisonnet654s83yes$3.741
2026-08-14 00:38tspattern_ismatchingBenzideepseek772s70yes$0.310
2026-08-14 00:42tspattern_ismatchingDeepSeek Harnessdeepseek225s26yes$0.053
yamlcpp_base64
2026-07-26 17:53yamlcpp_base64Claude Codesonnet88s15yes$0.403
2026-07-26 17:55yamlcpp_base64Benzideepseek117s13no$0.100
2026-07-26 17:58yamlcpp_base64Claude Codesonnet130s23yes$0.617
2026-07-26 18:01yamlcpp_base64Benzideepseek117s16no$0.122
2026-08-10 23:46yamlcpp_base64Benzideepseek269s19yes$0.057
2026-08-11 07:24yamlcpp_base64Benzisonnet134s11yes$0.207
2026-08-11 11:37yamlcpp_base64Claude Codesonnet60s13yes$0.379
2026-08-11 12:57yamlcpp_base64Claude Codesonnet44s11yes$0.312
2026-08-11 12:59yamlcpp_base64Benzisonnet86s10yes$0.194
2026-08-11 13:02yamlcpp_base64Benzideepseek172s27yes$0.037
2026-08-11 13:11yamlcpp_base64Claude Codesonnet54s15yes$0.396
2026-08-11 13:12yamlcpp_base64Benzisonnet77s11yes$0.211
2026-08-11 13:14yamlcpp_base64Benzideepseek120s15yes$0.016
2026-08-11 16:56yamlcpp_base64Claude Codesonnet74s10yes$0.361
2026-08-11 16:57yamlcpp_base64Benzisonnet69s11yes$0.324
2026-08-11 17:39yamlcpp_base64OpenCode—295s33yes$0.058
2026-08-11 17:41yamlcpp_base64Benzideepseek118s24yes$0.024
2026-08-11 19:35yamlcpp_base64Claude Codesonnet92s11yes$0.370
2026-08-11 19:37yamlcpp_base64Benzisonnet91s13yes$0.224
2026-08-12 16:46yamlcpp_base64Benzisonnet62s9yes$0.184
2026-08-12 17:11yamlcpp_base64Benzisonnet78s14yes$0.258
2026-08-12 18:13yamlcpp_base64Benzisonnet85s13yes$0.229
2026-08-12 21:08yamlcpp_base64Benzisonnet83s15yes$0.249
2026-08-12 22:01yamlcpp_base64Benzideepseek196s28yes$0.040
2026-08-13 10:32yamlcpp_base64Benzigpt110s9yes$0.104
2026-08-13 10:34yamlcpp_base64Codex CLIgpt146s12yes$0.205
2026-08-13 20:10yamlcpp_base64Benzideepseek224s36yes$0.068
2026-08-13 20:13yamlcpp_base64DeepSeek Harnessdeepseek154s16yes$0.024
2026-08-14 06:52yamlcpp_base64Codex CLIgpt30s8yes$0.123
zod_pipe_payload_flag
2026-08-07 05:14zod_pipe_payload_flagClaude Codesonnet3s1no—
2026-08-07 05:24zod_pipe_payload_flagBenzideepseek593s53yes$0.179
2026-08-07 05:49zod_pipe_payload_flagClaude Codesonnet180s41no$1.119
2026-08-07 08:43zod_pipe_payload_flagClaude Codesonnet190s41no$1.195
2026-08-07 09:59zod_pipe_payload_flagClaude Codesonnet741s72yes$3.478
2026-08-07 15:01zod_pipe_payload_flagBenzisonnet534s56yes$2.149
2026-08-11 00:56zod_pipe_payload_flagBenzideepseek966s92yes$0.382
2026-08-11 06:52zod_pipe_payload_flagBenzisonnet655s66yes$1.920
2026-08-12 02:15zod_pipe_payload_flagClaude Codesonnet438s58yes$2.473
2026-08-12 02:25zod_pipe_payload_flagBenzisonnet581s64yes$2.244
2026-08-12 02:35zod_pipe_payload_flagOpenCodesonnet523s78no$2.047
2026-08-12 18:54zod_pipe_payload_flagBenzisonnet339s53yes$1.779
2026-08-13 11:45zod_pipe_payload_flagBenzigpt58s10no$0.148
2026-08-13 11:51zod_pipe_payload_flagCodex CLIgpt329s41yes$0.673
2026-08-13 15:49zod_pipe_payload_flagBenzigpt928s80no$1.490
2026-08-13 17:54zod_pipe_payload_flagBenzisonnet508s68yes$1.850
2026-08-13 18:33zod_pipe_payload_flagBenzisonnet421s51yes$1.529
2026-08-13 19:00zod_pipe_payload_flagBenzigpt143s22yes$0.462
2026-08-13 19:12zod_pipe_payload_flagBenzigpt5257s10no$0.172
2026-08-13 21:43zod_pipe_payload_flagBenzideepseek595s49yes$0.133
2026-08-13 21:55zod_pipe_payload_flagDeepSeek Harnessdeepseek715s98yes$0.328
2026-08-14 07:07zod_pipe_payload_flagCodex CLIgpt141s19yes$0.424

Runs before 2026-08-11 were made on earlier versions of the Benzi engine and are kept for completeness, not comparison — the engine changed underneath them. The summary page uses the most recent run per arm per bug on the current engine.

SWE-bench Verified — every run

A separate benchmark from the 24-bug harness comparison above: the full SWE-bench Verified set, 500 real GitHub issues from twelve Python repositories, run once each through Benzi on DeepSeek v4-flash and graded by the official SWE-bench harness inside its own per-instance Docker images. Resolved 387 of 500 for $28.92 total. The summary view is on the SWE-bench page.

Network access to GitHub and PyPI was blocked inside every container, and no wall-clock timeout was applied to any instance. Three instances produced no patch at all — the agent investigated and stopped without attempting an edit — and are counted as failures. Seven instances were generated twice after faults in our own harness destroyed the first run; they were re-run on the basis of the fault, never the result.

On the wall times in this table. They are not comparable with the 24-bug run log above — that ran on different hardware entirely — and they are only loosely comparable with each other, because this run was itself split across two different machines. Instances that happened to start at the same moment also contended for the same cores, so an identical task could take 45 minutes in one slot and 10 in another. Treat these numbers as a record of what happened, not as a measure of how fast the agent is. None of it touches the result. SWE-bench scores whether the patch makes the tests pass; it is untimed, every instance ran to completion with no wall-clock cap, and grading used the identical official harness throughout. Turn counts, tokens, lines read and cost are hardware-independent and comparable everywhere.

SWE-bench Verified · every attempt recorded
instancerepomodelwallturnslines readresolvedcost
astropy
astropy-12907astropydeepseek1,058s9467yes$0.020
astropy-13033astropydeepseek919s29201no$0.052
astropy-13236astropydeepseek1,317s41861yes$0.095
astropy-13398astropydeepseek2,249s481,328no$0.152
astropy-13453astropydeepseek1,392s21524yes$0.055
astropy-13579astropydeepseek1,784s36741yes$0.087
astropy-13977astropydeepseek1,642s21280no$0.046
astropy-14096astropydeepseek2,230s24118yes$0.056
astropy-14182astropydeepseek824s28357no$0.040
astropy-14309astropydeepseek1,228s28148yes$0.039
astropy-14365astropydeepseek1,172s18288no$0.042
astropy-14369astropydeepseek1,424s441,030no$0.145
astropy-14508astropydeepseek1,486s34309yes$0.068
astropy-14539astropydeepseek892s12442yes$0.028
astropy-14598astropydeepseek518s39591no$0.098
astropy-14995astropydeepseek945s12383yes$0.023
astropy-7166astropydeepseek913s957yes$0.012
astropy-7336astropydeepseek952s14103yes$0.015
astropy-7606astropydeepseek219s14182yes$0.016
astropy-7671astropydeepseek1,212s17516yes$0.031
astropy-8707astropydeepseek1,874s23343yes$0.045
astropy-8872astropydeepseek1,605s17138yes$0.023
django
django-10097djangodeepseek1,494s29145no$0.074
django-10554djangodeepseek3,360s121598no$0.634
django-10880djangodeepseek699s23114yes$0.036
django-10914djangodeepseek564s10189yes$0.016
django-10973djangodeepseek1,022s14187no$0.025
django-10999djangodeepseek401s632no$0.007
django-11066djangodeepseek402s637yes$0.007
django-11087djangodeepseek1,869s49885no$0.179
django-11095djangodeepseek484s1440yes$0.012
django-11099djangodeepseek492s1314yes$0.012
django-11119djangodeepseek534s13109yes$0.015
django-11133djangodeepseek500s13201yes$0.018
django-11138djangodeepseek2,584s641,864yes$0.233
django-11141djangodeepseek864s25313no$0.057
django-11149djangodeepseek566s16577no$0.033
django-11163djangodeepseek363s1223yes$0.008
django-11179djangodeepseek418s10131yes$0.010
django-11206djangodeepseek880s39286yes$0.100
django-11211djangodeepseek712s37254yes$0.059
django-11239djangodeepseek353s10217no$0.010
django-11265djangodeepseek1,237s431,255yes$0.120
django-11276djangodeepseek788s23179yes$0.047
django-11292djangodeepseek766s24222yes$0.031
django-11299djangodeepseek444s18172yes$0.017
django-11333djangodeepseek519s17285yes$0.022
django-11400djangodeepseek857s35412yes$0.070
django-11433djangodeepseek581s1168yes$0.029
django-11451djangodeepseek366s862yes$0.007
django-11477djangodeepseek638s30705no$0.042
django-11490djangodeepseek700s22221yes$0.037
django-11532djangodeepseek491s13124no$0.024
django-11551djangodeepseek543s18112yes$0.023
django-11555djangodeepseek658s23195yes$0.046
django-11603djangodeepseek596s19208yes$0.034
django-11728djangodeepseek641s19149yes$0.034
django-11734djangodeepseek1,804s——no$0.000
django-11740djangodeepseek553s24385yes$0.035
django-11749djangodeepseek681s19182yes$0.027
django-11790djangodeepseek1,252s65359yes$0.154
django-11815djangodeepseek718s1966yes$0.025
django-11820djangodeepseek288s1192no$0.013
django-11848djangodeepseek584s1184no$0.013
django-11880djangodeepseek474s17180yes$0.017
django-11885djangodeepseek1,147s35652no$0.108
django-11951djangodeepseek484s1063yes$0.013
django-11964djangodeepseek779s17352yes$0.051
django-11999djangodeepseek690s19401yes$0.045
django-12039djangodeepseek607s17381yes$0.036
django-12050djangodeepseek391s1546yes$0.022
django-12125djangodeepseek649s30288yes$0.045
django-12143djangodeepseek313s124yes$0.011
django-12155djangodeepseek580s13155yes$0.020
django-12193djangodeepseek658s30147yes$0.045
django-12209djangodeepseek1,304s32277yes$0.097
django-12262djangodeepseek510s18356yes$0.022
django-12273djangodeepseek992s32319no$0.080
django-12276djangodeepseek367s9119yes$0.011
django-12304djangodeepseek560s19187yes$0.034
django-12308djangodeepseek581s17299yes$0.024
django-12325djangodeepseek565s15232no$0.025
django-12406djangodeepseek819s21452no$0.070
django-12419djangodeepseek1,078s35432yes$0.077
django-12663djangodeepseek2,172s71217yes$0.242
django-12708djangodeepseek624s18148yes$0.036
django-12713djangodeepseek709s18144yes$0.034
django-12741djangodeepseek280s882yes$0.008
django-12754djangodeepseek954s51978yes$0.107
django-12774djangodeepseek537s17107no$0.015
django-12858djangodeepseek1,546s——yes$0.000
django-12965djangodeepseek950s29633yes$0.086
django-13012djangodeepseek668s21280yes$0.045
django-13023djangodeepseek424s12268yes$0.015
django-13028djangodeepseek404s1460yes$0.015
django-13033djangodeepseek721s21512yes$0.062
django-13089djangodeepseek468s1061yes$0.016
django-13109djangodeepseek484s1547yes$0.017
django-13112djangodeepseek1,041s47970yes$0.116
django-13121djangodeepseek1,023s57689yes$0.164
django-13128djangodeepseek1,604s30201yes$0.059
django-13158djangodeepseek985s34387yes$0.068
django-13195djangodeepseek575s19440no$0.030
django-13212djangodeepseek1,036s23690no$0.043
django-13279djangodeepseek461s18379yes$0.024
django-13297djangodeepseek840s19237no$0.061
django-13315djangodeepseek758s17242yes$0.049
django-13343djangodeepseek575s13235yes$0.019
django-13344djangodeepseek1,092s25898no$0.054
django-13346djangodeepseek783s17403yes$0.027
django-13363djangodeepseek534s13348yes$0.017
django-13401djangodeepseek847s22244yes$0.053
django-13406djangodeepseek1,800s71803no$0.192
django-13410djangodeepseek416s7115yes$0.010
django-13417djangodeepseek723s28464yes$0.057
django-13449djangodeepseek548s32393yes$0.089
django-13512djangodeepseek834s24324yes$0.040
django-13513djangodeepseek564s2275no$0.029
django-13516djangodeepseek446s14155yes$0.016
django-13551djangodeepseek429s9361yes$0.013
django-13568djangodeepseek616s20370yes$0.027
django-13569djangodeepseek686s23362yes$0.042
django-13590djangodeepseek680s1833yes$0.025
django-13658djangodeepseek409s1133yes$0.011
django-13670djangodeepseek347s1063yes$0.010
django-13741djangodeepseek412s12109yes$0.014
django-13786djangodeepseek478s12177yes$0.010
django-13794djangodeepseek566s18237no$0.038
django-13807djangodeepseek865s31162yes$0.043
django-13809djangodeepseek1,006s20467yes$0.036
django-13810djangodeepseek597s18507yes$0.031
django-13820djangodeepseek432s10163yes$0.014
django-13821djangodeepseek650s20195yes$0.040
django-13837djangodeepseek553s15168yes$0.021
django-13925djangodeepseek875s36358yes$0.086
django-13933djangodeepseek505s19342yes$0.021
django-13964djangodeepseek566s13172yes$0.019
django-14007djangodeepseek753s21267yes$0.048
django-14011djangodeepseek554s17386no$0.028
django-14017djangodeepseek793s25248yes$0.054
django-14034djangodeepseek791s25378no$0.049
django-14053djangodeepseek1,150s49444yes$0.099
django-14089djangodeepseek349s1073yes$0.009
django-14122djangodeepseek757s39824yes$0.077
django-14140djangodeepseek432s12236no$0.015
django-14155djangodeepseek797s27339no$0.047
django-14170djangodeepseek469s18207no$0.020
django-14238djangodeepseek517s17150yes$0.020
django-14311djangodeepseek590s10137yes$0.017
django-14315djangodeepseek444s11104no$0.017
django-14349djangodeepseek498s992yes$0.012
django-14351djangodeepseek1,148s39370yes$0.092
django-14373djangodeepseek439s1357yes$0.014
django-14376djangodeepseek684s15361no$0.024
django-14404djangodeepseek1,254s45198yes$0.062
django-14434djangodeepseek518s17253yes$0.023
django-14493djangodeepseek612s15129yes$0.021
django-14500djangodeepseek994s22765yes$0.064
django-14534djangodeepseek856s18422no$0.053
django-14539djangodeepseek911s12520yes$0.046
django-14559djangodeepseek397s1062no$0.014
django-14580djangodeepseek522s14378yes$0.021
django-14608djangodeepseek539s16216yes$0.021
django-14631djangodeepseek920s27871yes$0.063
django-14672djangodeepseek578s1342yes$0.011
django-14725djangodeepseek2,175s39318yes$0.128
django-14752djangodeepseek535s8105yes$0.007
django-14765djangodeepseek459s910yes$0.007
django-14771djangodeepseek691s12156yes$0.034
django-14787djangodeepseek424s8120yes$0.012
django-14792djangodeepseek1,047s34444no$0.096
django-14855djangodeepseek1,103s56618yes$0.079
django-14915djangodeepseek461s883yes$0.009
django-14999djangodeepseek1,016s25632yes$0.065
django-15022djangodeepseek1,471s31789yes$0.095
django-15037djangodeepseek1,078s26688no$0.082
django-15098djangodeepseek1,684s601,081no$0.177
django-15103djangodeepseek506s1360yes$0.015
django-15104djangodeepseek466s1374yes$0.014
django-15127djangodeepseek520s12219no$0.013
django-15128djangodeepseek1,360s37294yes$0.086
django-15161djangodeepseek1,503s511,137yes$0.150
django-15252djangodeepseek1,044s441,012no$0.175
django-15268djangodeepseek747s23558yes$0.056
django-15277djangodeepseek568s16224yes$0.018
django-15278djangodeepseek600s17223yes$0.029
django-15280djangodeepseek1,288s431,104yes$0.141
django-15315djangodeepseek709s22165yes$0.032
django-15368djangodeepseek711s18176yes$0.024
django-15375djangodeepseek868s17466yes$0.057
django-15380djangodeepseek547s20168yes$0.021
django-15382djangodeepseek669s35343yes$0.051
django-15467djangodeepseek460s1137yes$0.013
django-15499djangodeepseek597s14349yes$0.020
django-15503djangodeepseek1,821s56871yes$0.138
django-15525djangodeepseek1,004s211,220yes$0.052
django-15554djangodeepseek3,189s56770yes$0.161
django-15561djangodeepseek890s20591yes$0.051
django-15563djangodeepseek1,312s38660no$0.101
django-15569djangodeepseek668s22124yes$0.025
django-15572djangodeepseek857s22977yes$0.049
django-15629djangodeepseek1,576s601,527no$0.194
django-15695djangodeepseek996s27503no$0.067
django-15731djangodeepseek697s13213yes$0.017
django-15732djangodeepseek1,188s36433yes$0.079
django-15741djangodeepseek517s1966yes$0.019
django-15814djangodeepseek948s29718yes$0.067
django-15851djangodeepseek540s9219yes$0.010
django-15863djangodeepseek721s11260yes$0.020
django-15916djangodeepseek709s12243no$0.025
django-15930djangodeepseek874s24505yes$0.041
django-15957djangodeepseek2,025s601,036no$0.184
django-15973djangodeepseek2,095s70560no$0.208
django-15987djangodeepseek829s15175no$0.021
django-16032djangodeepseek1,584s78602yes$0.145
django-16082djangodeepseek662s19551yes$0.024
django-16100djangodeepseek555s9217yes$0.012
django-16116djangodeepseek793s26866yes$0.050
django-16136djangodeepseek721s10299yes$0.025
django-16139djangodeepseek454s1237yes$0.011
django-16145djangodeepseek591s21421yes$0.029
django-16255djangodeepseek480s1328yes$0.012
django-16256djangodeepseek1,844s74994no$0.200
django-16263djangodeepseek4,141s72779no$0.240
django-16315djangodeepseek908s36478yes$0.095
django-16333djangodeepseek690s1584yes$0.017
django-16429djangodeepseek472s9137yes$0.013
django-16454djangodeepseek996s2573no$0.039
django-16485djangodeepseek914s15111yes$0.024
django-16493djangodeepseek456s1131yes$0.010
django-16502djangodeepseek1,048s——no$0.000
django-16527djangodeepseek433s9195yes$0.010
django-16560djangodeepseek1,985s351,165yes$0.080
django-16569djangodeepseek505s1036yes$0.011
django-16595djangodeepseek715s23464yes$0.036
django-16612djangodeepseek457s12202yes$0.013
django-16631djangodeepseek1,076s16628no$0.056
django-16642djangodeepseek494s11372yes$0.012
django-16661djangodeepseek2,051s59293yes$0.113
django-16662djangodeepseek620s15160yes$0.018
django-16667djangodeepseek587s1798no$0.015
django-16801djangodeepseek436s9121yes$0.012
django-16819djangodeepseek1,007s29761yes$0.070
django-16877djangodeepseek672s9168yes$0.011
django-16899djangodeepseek834s16147yes$0.021
django-16901djangodeepseek891s20183yes$0.036
django-16938djangodeepseek1,460s29345yes$0.069
django-16950djangodeepseek1,799s44902yes$0.167
django-17029djangodeepseek728s1446yes$0.014
django-17084djangodeepseek1,843s52317yes$0.158
django-17087djangodeepseek812s13115yes$0.017
django-7530djangodeepseek907s1786yes$0.021
django-9296djangodeepseek376s899yes$0.007
flask
flask-5014flaskdeepseek294s1484yes$0.015
matplotlib
matplotlib-13989matplotlibdeepseek854s1747yes$0.022
matplotlib-14623matplotlibdeepseek1,165s14427no$0.029
matplotlib-20488matplotlibdeepseek2,812s30632yes$0.101
matplotlib-20676matplotlibdeepseek2,348s28922yes$0.061
matplotlib-20826matplotlibdeepseek2,167s32599yes$0.062
matplotlib-20859matplotlibdeepseek2,383s22654yes$0.030
matplotlib-21568matplotlibdeepseek5,838s80553no$0.206
matplotlib-22719matplotlibdeepseek2,609s16101yes$0.040
matplotlib-22865matplotlibdeepseek4,168s21487yes$0.042
matplotlib-22871matplotlibdeepseek4,590s21357yes$0.032
matplotlib-23299matplotlibdeepseek3,560s22558no$0.044
matplotlib-23314matplotlibdeepseek3,208s1490yes$0.018
matplotlib-23412matplotlibdeepseek3,321s23457yes$0.036
matplotlib-23476matplotlibdeepseek3,079s22177no$0.061
matplotlib-24026matplotlibdeepseek2,969s16186yes$0.031
matplotlib-24149matplotlibdeepseek1,170s16162yes$0.030
matplotlib-24177matplotlibdeepseek1,972s35343yes$0.056
matplotlib-24570matplotlibdeepseek1,580s22487yes$0.047
matplotlib-24627matplotlibdeepseek3,352s37576yes$0.083
matplotlib-24637matplotlibdeepseek1,840s20414yes$0.030
matplotlib-24870matplotlibdeepseek2,882s28511no$0.072
matplotlib-24970matplotlibdeepseek3,606s1977yes$0.036
matplotlib-25122matplotlibdeepseek3,745s1279yes$0.015
matplotlib-25287matplotlibdeepseek3,224s23273yes$0.038
matplotlib-25311matplotlibdeepseek4,060s59691yes$0.131
matplotlib-25332matplotlibdeepseek4,148s36450yes$0.048
matplotlib-25479matplotlibdeepseek2,190s22527no$0.035
matplotlib-25775matplotlibdeepseek3,477s35723yes$0.060
matplotlib-25960matplotlibdeepseek3,817s41825yes$0.111
matplotlib-26113matplotlibdeepseek2,346s17204yes$0.037
matplotlib-26208matplotlibdeepseek3,286s38501no$0.087
matplotlib-26291matplotlibdeepseek2,873s25389yes$0.050
matplotlib-26342matplotlibdeepseek1,936s12136yes$0.013
matplotlib-26466matplotlibdeepseek1,277s21509no$0.042
pylint
pylint-4551pylintdeepseek1,302s711,463no$0.182
pylint-4604pylintdeepseek457s25318no$0.044
pylint-4661pylintdeepseek438s15102no$0.032
pylint-4970pylintdeepseek1,546s721,717no$0.170
pylint-6386pylintdeepseek963s451,161yes$0.087
pylint-6528pylintdeepseek762s41567yes$0.088
pylint-6903pylintdeepseek586s20165yes$0.034
pylint-7080pylintdeepseek1,465s831,021no$0.273
pylint-7277pylintdeepseek319s12188yes$0.019
pylint-8898pylintdeepseek769s26221no$0.046
pytest
pytest-10051pytestdeepseek588s17339yes$0.046
pytest-10081pytestdeepseek277s14356yes$0.032
pytest-10356pytestdeepseek1,027s55392no$0.151
pytest-5262pytestdeepseek366s22141yes$0.028
pytest-5631pytestdeepseek311s1234yes$0.013
pytest-5787pytestdeepseek762s36819yes$0.108
pytest-5809pytestdeepseek175s638yes$0.007
pytest-5840pytestdeepseek1,568s32484yes$0.133
pytest-6197pytestdeepseek3,349s67511yes$0.213
pytest-6202pytestdeepseek299s2488yes$0.043
pytest-7205pytestdeepseek574s22641yes$0.060
pytest-7236pytestdeepseek466s26652yes$0.075
pytest-7324pytestdeepseek2,313s39415yes$0.100
pytest-7432pytestdeepseek404s20534yes$0.043
pytest-7490pytestdeepseek497s32500yes$0.054
pytest-7521pytestdeepseek254s18175yes$0.041
pytest-7571pytestdeepseek317s26307yes$0.054
pytest-7982pytestdeepseek262s1133yes$0.012
pytest-8399pytestdeepseek385s30234yes$0.038
requests
requests-1142requestsdeepseek341s19452yes$0.034
requests-1724requestsdeepseek241s20425yes$0.044
requests-1766requestsdeepseek94s1146yes$0.014
requests-1921requestsdeepseek218s18169yes$0.030
requests-2317requestsdeepseek162s13214yes$0.020
requests-2931requestsdeepseek276s20281yes$0.036
requests-5414requestsdeepseek123s14127yes$0.021
requests-6028requestsdeepseek553s42452yes$0.117
scikit-learn
scikit-learn-10297scikit-learndeepseek2,249s16289yes$0.019
scikit-learn-10844scikit-learndeepseek2,110s8103yes$0.012
scikit-learn-10908scikit-learndeepseek1,993s15128yes$0.017
scikit-learn-11310scikit-learndeepseek2,602s23256yes$0.033
scikit-learn-11578scikit-learndeepseek2,235s20552yes$0.041
scikit-learn-12585scikit-learndeepseek2,019s1697yes$0.022
scikit-learn-12682scikit-learndeepseek4,159s28913yes$0.045
scikit-learn-12973scikit-learndeepseek1,918s17132yes$0.019
scikit-learn-13124scikit-learndeepseek1,693s10519yes$0.022
scikit-learn-13135scikit-learndeepseek2,711s21306yes$0.052
scikit-learn-13142scikit-learndeepseek1,772s9126yes$0.011
scikit-learn-13328scikit-learndeepseek1,972s11733yes$0.018
scikit-learn-13439scikit-learndeepseek1,751s1269yes$0.015
scikit-learn-13496scikit-learndeepseek2,327s15780yes$0.023
scikit-learn-13779scikit-learndeepseek2,268s1263yes$0.012
scikit-learn-14053scikit-learndeepseek1,965s11204yes$0.015
scikit-learn-14087scikit-learndeepseek3,025s25526yes$0.049
scikit-learn-14141scikit-learndeepseek2,032s955yes$0.015
scikit-learn-14496scikit-learndeepseek2,559s1271yes$0.031
scikit-learn-14629scikit-learndeepseek2,230s18549yes$0.038
scikit-learn-14710scikit-learndeepseek7,348s40583no$0.075
scikit-learn-14894scikit-learndeepseek2,330s13169yes$0.027
scikit-learn-14983scikit-learndeepseek3,007s38307yes$0.063
scikit-learn-15100scikit-learndeepseek2,158s9103yes$0.015
scikit-learn-25102scikit-learndeepseek7,524s——yes$0.000
scikit-learn-25232scikit-learndeepseek1,824s39696yes$0.071
scikit-learn-25747scikit-learndeepseek6,820s——no$0.000
scikit-learn-25931scikit-learndeepseek6,735s——yes$0.000
scikit-learn-25973scikit-learndeepseek5,771s——yes$0.000
scikit-learn-26194scikit-learndeepseek5,689s——no$0.000
scikit-learn-26323scikit-learndeepseek1,416s26461yes$0.041
scikit-learn-9288scikit-learndeepseek596s20735yes$0.063
seaborn
seaborn-3069seaborndeepseek1,887s451,129yes$0.153
seaborn-3187seaborndeepseek1,160s36700no$0.094
sphinx
sphinx-10323sphinxdeepseek1,200s34339yes$0.062
sphinx-10435sphinxdeepseek1,154s17293yes$0.037
sphinx-10449sphinxdeepseek1,271s31727yes$0.073
sphinx-10466sphinxdeepseek846s17600no$0.029
sphinx-10614sphinxdeepseek1,178s37635no$0.082
sphinx-10673sphinxdeepseek1,444s451,265yes$0.131
sphinx-11445sphinxdeepseek870s1854yes$0.026
sphinx-11510sphinxdeepseek1,840s40477no$0.113
sphinx-7440sphinxdeepseek1,554s64626yes$0.143
sphinx-7454sphinxdeepseek888s30298yes$0.057
sphinx-7462sphinxdeepseek518s1790no$0.030
sphinx-7590sphinxdeepseek901s31747no$0.076
sphinx-7748sphinxdeepseek835s45821no$0.089
sphinx-7757sphinxdeepseek549s29225yes$0.055
sphinx-7889sphinxdeepseek523s16280yes$0.028
sphinx-7910sphinxdeepseek766s25273yes$0.043
sphinx-7985sphinxdeepseek1,474s37618yes$0.103
sphinx-8035sphinxdeepseek939s29587yes$0.058
sphinx-8056sphinxdeepseek893s51680no$0.092
sphinx-8120sphinxdeepseek1,072s37320yes$0.125
sphinx-8265sphinxdeepseek658s32689yes$0.076
sphinx-8269sphinxdeepseek406s14751yes$0.022
sphinx-8459sphinxdeepseek636s25306yes$0.035
sphinx-8475sphinxdeepseek470s13677yes$0.017
sphinx-8548sphinxdeepseek4,514s1101,142no$0.385
sphinx-8551sphinxdeepseek560s26641yes$0.058
sphinx-8593sphinxdeepseek679s23507yes$0.038
sphinx-8595sphinxdeepseek640s15138yes$0.019
sphinx-8621sphinxdeepseek1,126s21119yes$0.040
sphinx-8638sphinxdeepseek5,191s1271,579no$1.209
sphinx-8721sphinxdeepseek603s17302yes$0.028
sphinx-9229sphinxdeepseek2,345s1281,498no$0.486
sphinx-9230sphinxdeepseek918s37686yes$0.093
sphinx-9258sphinxdeepseek720s42678yes$0.099
sphinx-9281sphinxdeepseek399s14212yes$0.018
sphinx-9320sphinxdeepseek392s21369yes$0.024
sphinx-9367sphinxdeepseek282s656yes$0.007
sphinx-9461sphinxdeepseek973s24838no$0.071
sphinx-9591sphinxdeepseek475s271,095yes$0.053
sphinx-9602sphinxdeepseek685s33338no$0.053
sphinx-9658sphinxdeepseek658s38593yes$0.073
sphinx-9673sphinxdeepseek577s20309yes$0.033
sphinx-9698sphinxdeepseek354s9251yes$0.012
sphinx-9711sphinxdeepseek509s17119no$0.034
sympy
sympy-11618sympydeepseek1,087s21174yes$0.022
sympy-12096sympydeepseek1,087s16104yes$0.021
sympy-12419sympydeepseek2,861s57484yes$0.141
sympy-12481sympydeepseek1,518s26462yes$0.058
sympy-12489sympydeepseek2,567s31336yes$0.089
sympy-13031sympydeepseek1,382s34273yes$0.060
sympy-13091sympydeepseek8,418s61534yes$0.252
sympy-13372sympydeepseek13,348s15235yes$0.023
sympy-13480sympydeepseek13,904s10101yes$0.012
sympy-13551sympydeepseek3,291s24492yes$0.052
sympy-13615sympydeepseek13,281s17326yes$0.029
sympy-13647sympydeepseek3,432s2296yes$0.025
sympy-13757sympydeepseek5,248s741,215yes$0.249
sympy-13798sympydeepseek2,973s21184no$0.027
sympy-13852sympydeepseek3,765s33296no$0.068
sympy-13877sympydeepseek3,008s28336yes$0.048
sympy-13878sympydeepseek4,458s50617yes$0.113
sympy-13974sympydeepseek2,736s28686no$0.075
sympy-14248sympydeepseek5,049s691,339no$0.346
sympy-14531sympydeepseek3,407s26299yes$0.063
sympy-14711sympydeepseek3,014s12111no$0.018
sympy-14976sympydeepseek3,345s18132yes$0.021
sympy-15017sympydeepseek619s24278yes$0.026
sympy-15345sympydeepseek615s16375yes$0.029
sympy-15349sympydeepseek559s15130yes$0.021
sympy-15599sympydeepseek1,322s35353yes$0.102
sympy-15809sympydeepseek564s15212yes$0.018
sympy-15875sympydeepseek902s3962no$0.063
sympy-15976sympydeepseek901s27516yes$0.066
sympy-16450sympydeepseek738s24165yes$0.030
sympy-16597sympydeepseek1,150s28722no$0.059
sympy-16766sympydeepseek547s15291yes$0.019
sympy-16792sympydeepseek918s42478yes$0.090
sympy-16886sympydeepseek475s830no$0.009
sympy-17139sympydeepseek573s870yes$0.012
sympy-17318sympydeepseek828s17346yes$0.040
sympy-17630sympydeepseek1,559s681,354yes$0.173
sympy-17655sympydeepseek1,379s43473yes$0.116
sympy-18189sympydeepseek633s15251yes$0.018
sympy-18199sympydeepseek693s24214no$0.037
sympy-18211sympydeepseek1,157s32540yes$0.089
sympy-18698sympydeepseek1,766s62710yes$0.171
sympy-18763sympydeepseek633s1865no$0.026
sympy-19040sympydeepseek2,372s78952yes$0.196
sympy-19346sympydeepseek708s21425yes$0.036
sympy-19495sympydeepseek720s18426yes$0.036
sympy-19637sympydeepseek553s16129yes$0.015
sympy-19783sympydeepseek829s22733yes$0.052
sympy-19954sympydeepseek546s8127yes$0.012
sympy-20154sympydeepseek725s20392yes$0.037
sympy-20428sympydeepseek1,182s33395no$0.074
sympy-20438sympydeepseek1,153s411,404no$0.113
sympy-20590sympydeepseek636s23150yes$0.035
sympy-20801sympydeepseek841s31138yes$0.066
sympy-20916sympydeepseek781s29411no$0.041
sympy-21379sympydeepseek1,583s47967yes$0.083
sympy-21596sympydeepseek1,229s30557no$0.088
sympy-21612sympydeepseek1,213s40811yes$0.097
sympy-21847sympydeepseek703s17138yes$0.024
sympy-21930sympydeepseek991s27161no$0.060
sympy-22080sympydeepseek1,094s46875no$0.123
sympy-22456sympydeepseek2,232s57852yes$0.192
sympy-22714sympydeepseek964s29456yes$0.049
sympy-22914sympydeepseek558s1249yes$0.012
sympy-23262sympydeepseek587s1928yes$0.028
sympy-23413sympydeepseek2,499s27504no$0.137
sympy-23534sympydeepseek584s14242yes$0.016
sympy-23824sympydeepseek574s15384yes$0.017
sympy-23950sympydeepseek1,091s44306yes$0.096
sympy-24066sympydeepseek1,026s32387yes$0.052
sympy-24213sympydeepseek605s19451yes$0.025
sympy-24443sympydeepseek521s17942yes$0.043
sympy-24539sympydeepseek332s1864yes$0.020
sympy-24562sympydeepseek406s14123yes$0.021
sympy-24661sympydeepseek416s23360yes$0.038
xarray
xarray-2905xarraydeepseek1,885s31462yes$0.101
xarray-3095xarraydeepseek875s17217yes$0.024
xarray-3151xarraydeepseek697s13237yes$0.016
xarray-3305xarraydeepseek663s17407yes$0.024
xarray-3677xarraydeepseek550s16428yes$0.020
xarray-3993xarraydeepseek528s12216no$0.016
xarray-4075xarraydeepseek400s8221yes$0.019
xarray-4094xarraydeepseek328s20233yes$0.026
xarray-4356xarraydeepseek940s24371yes$0.057
xarray-4629xarraydeepseek226s1232yes$0.021
xarray-4687xarraydeepseek1,022s46906no$0.129
xarray-4695xarraydeepseek474s20190yes$0.028
xarray-4966xarraydeepseek571s32345yes$0.050
xarray-6461xarraydeepseek1,231s26334yes$0.049
xarray-6599xarraydeepseek1,522s51556yes$0.101
xarray-6721xarraydeepseek995s17117yes$0.038
xarray-6744xarraydeepseek1,113s29364yes$0.050
xarray-6938xarraydeepseek1,008s26371no$0.036
xarray-6992xarraydeepseek1,026s11344no$0.029
xarray-7229xarraydeepseek1,282s29641no$0.085
xarray-7233xarraydeepseek988s26283yes$0.033
xarray-7393xarraydeepseek491s20453yes$0.025
deepseek v4-pro · 33 instances
astropy-14365astropydeepseek-pro———no—
django-11149djangodeepseek-pro———yes—
django-11239djangodeepseek-pro———yes—
django-11532djangodeepseek-pro———no—
django-11848djangodeepseek-pro———no—
django-11885djangodeepseek-pro———yes—
django-12273djangodeepseek-pro———no—
django-12774djangodeepseek-pro———yes—
django-13297djangodeepseek-pro———yes—
django-13344djangodeepseek-pro———yes—
django-13406djangodeepseek-pro———yes—
django-14376djangodeepseek-pro———no—
django-14534djangodeepseek-pro———no—
django-14559djangodeepseek-pro———yes—
django-15037djangodeepseek-pro———yes—
django-15127djangodeepseek-pro———no—
django-15916djangodeepseek-pro———no—
django-15957djangodeepseek-pro———yes—
django-15973djangodeepseek-pro———yes—
django-15987djangodeepseek-pro———yes—
django-16454djangodeepseek-pro———yes—
matplotlib-14623matplotlibdeepseek-pro———yes—
matplotlib-24870matplotlibdeepseek-pro———no—
xarray-3993xarraydeepseek-pro———yes—
xarray-4687xarraydeepseek-pro———yes—
xarray-6938xarraydeepseek-pro———no—
pytest-10356pytestdeepseek-pro———no—
scikit-learn-14710scikit-learndeepseek-pro———yes—
sphinx-10466sphinxdeepseek-pro———yes—
sphinx-8056sphinxdeepseek-pro———no—
sphinx-8638sphinxdeepseek-pro———no—
sympy-16886sympydeepseek-pro———no—
sympy-20916sympydeepseek-pro———no—
flash + unannounced escalation tool · 20 instances
astropy-14598astropydeepseek———no—
astropy-7606astropydeepseek———yes—
django-11477djangodeepseek———no—
django-12325djangodeepseek———no—
django-13195djangodeepseek———no—
django-14140djangodeepseek———yes—
django-14170djangodeepseek———no—
requests-1724psfdeepseek———no—
requests-1766psfdeepseek———no—
requests-1921psfdeepseek———no—
requests-2317psfdeepseek———no—
requests-2931psfdeepseek———yes—
requests-5414psfdeepseek———yes—
xarray-4629xarraydeepseek———yes—
pylint-8898pylintdeepseek———yes—
pytest-5809pytestdeepseek———yes—
pytest-7521pytestdeepseek———yes—
pytest-7982pytestdeepseek———yes—
sphinx-9602sphinxdeepseek———no—
sympy-18199sympydeepseek———no—
deepseek v4-pro · 89 instances
astropy-12907astropydeepseek-pro———yes—
astropy-13236astropydeepseek-pro———no—
astropy-13398astropydeepseek-pro———no—
astropy-13453astropydeepseek-pro———yes—
astropy-13579astropydeepseek-pro———yes—
astropy-13977astropydeepseek-pro———no—
astropy-14096astropydeepseek-pro———yes—
astropy-14309astropydeepseek-pro———yes—
astropy-14369astropydeepseek-pro———no—
astropy-14508astropydeepseek-pro———yes—
astropy-14539astropydeepseek-pro———yes—
astropy-14598astropydeepseek-pro———no—
astropy-14995astropydeepseek-pro———yes—
astropy-7166astropydeepseek-pro———yes—
astropy-7336astropydeepseek-pro———yes—
astropy-7606astropydeepseek-pro———yes—
astropy-7671astropydeepseek-pro———yes—
astropy-8707astropydeepseek-pro———yes—
astropy-8872astropydeepseek-pro———yes—
django-10097djangodeepseek-pro———no—
django-10554djangodeepseek-pro———no—
django-10880djangodeepseek-pro———yes—
django-10914djangodeepseek-pro———yes—
django-10973djangodeepseek-pro———no—
django-10999djangodeepseek-pro———no—
django-11066djangodeepseek-pro———yes—
django-11087djangodeepseek-pro———yes—
django-11095djangodeepseek-pro———yes—
django-11099djangodeepseek-pro———yes—
django-11119djangodeepseek-pro———yes—
django-11133djangodeepseek-pro———yes—
django-11138djangodeepseek-pro———yes—
django-11141djangodeepseek-pro———no—
django-11163djangodeepseek-pro———yes—
django-11179djangodeepseek-pro———yes—
django-11206djangodeepseek-pro———yes—
django-11211djangodeepseek-pro———yes—
django-11265djangodeepseek-pro———yes—
django-11276djangodeepseek-pro———yes—
django-11292djangodeepseek-pro———yes—
django-11299djangodeepseek-pro———yes—
django-11333djangodeepseek-pro———yes—
django-11400djangodeepseek-pro———no—
django-11433djangodeepseek-pro———yes—
django-11451djangodeepseek-pro———yes—
django-11477djangodeepseek-pro———yes—
django-11490djangodeepseek-pro———yes—
django-11551djangodeepseek-pro———yes—
django-11555djangodeepseek-pro———yes—
django-11603djangodeepseek-pro———yes—
django-11728djangodeepseek-pro———no—
django-11734djangodeepseek-pro———no—
django-11740djangodeepseek-pro———yes—
django-11749djangodeepseek-pro———yes—
django-11790djangodeepseek-pro———yes—
django-11815djangodeepseek-pro———yes—
django-11820djangodeepseek-pro———no—
django-11880djangodeepseek-pro———yes—
django-11951djangodeepseek-pro———yes—
django-11964djangodeepseek-pro———no—
django-11999djangodeepseek-pro———yes—
django-12039djangodeepseek-pro———yes—
django-12050djangodeepseek-pro———yes—
django-12143djangodeepseek-pro———yes—
django-12155djangodeepseek-pro———yes—
django-12193djangodeepseek-pro———no—
django-12262djangodeepseek-pro———yes—
django-12276djangodeepseek-pro———yes—
django-12308djangodeepseek-pro———no—
django-12325djangodeepseek-pro———no—
django-12406djangodeepseek-pro———no—
django-12708djangodeepseek-pro———yes—
django-12713djangodeepseek-pro———yes—
django-12741djangodeepseek-pro———yes—
django-13195djangodeepseek-pro———no—
django-13212djangodeepseek-pro———no—
django-13794djangodeepseek-pro———no—
django-14011djangodeepseek-pro———no—
django-14034djangodeepseek-pro———no—
django-14140djangodeepseek-pro———yes—
django-14155djangodeepseek-pro———no—
django-14170djangodeepseek-pro———no—
django-14315djangodeepseek-pro———no—
django-15629djangodeepseek-pro———no—
django-15695djangodeepseek-pro———no—
django-16256djangodeepseek-pro———no—
django-16631djangodeepseek-pro———no—
django-16667djangodeepseek-pro———no—
matplotlib-21568matplotlibdeepseek-pro———no—
flash + scope cards · 69 failures re-attempted
astropy-13033astropydeepseek1,266s18228no—
astropy-13398astropydeepseek1,713s39822no—
astropy-13977astropydeepseek1,587s46703no—
astropy-14182astropydeepseek1,338s31417yes—
astropy-14365astropydeepseek1,243s19446yes—
astropy-14369astropydeepseek1,574s34970no—
astropy-14598astropydeepseek1,663s38720no—
astropy-7606astropydeepseek1,160s16135yes—
django-10097djangodeepseek1,405s38257no—
django-10554djangodeepseek3,202s811,247no—
django-10973djangodeepseek1,159s15222no—
django-10999djangodeepseek1,027s1042no—
django-11087djangodeepseek1,153s671,198no—
django-11141djangodeepseek655s38323no—
django-11149djangodeepseek560s36647yes—
django-11239djangodeepseek376s22409no—
django-11532djangodeepseek340s23353no—
django-11734djangodeepseek1,099s74618no—
django-11820djangodeepseek270s19245no—
django-11848djangodeepseek197s13156no—
django-11885djangodeepseek1,298s42769no—
django-12273djangodeepseek711s58588no—
django-12325djangodeepseek447s25193no—
django-12406djangodeepseek545s23384yes—
django-12774djangodeepseek292s17156yes—
django-13195djangodeepseek267s20107no—
django-13212djangodeepseek364s30624no—
django-13297djangodeepseek461s23255no—
django-13344djangodeepseek1,066s58718yes—
django-13406djangodeepseek1,103s661,373no—
django-14011djangodeepseek662s44364no—
django-14140djangodeepseek263s19364no—
django-14155djangodeepseek425s27261no—
django-14315djangodeepseek195s23171no—
django-14376djangodeepseek291s24297no—
django-14534djangodeepseek622s34149no—
django-14559djangodeepseek210s15322no—
django-15037djangodeepseek431s34530no—
django-15098djangodeepseek854s51925no—
django-15563djangodeepseek787s43473yes—
django-15629djangodeepseek752s54819no—
django-15695djangodeepseek630s57597no—
django-15916djangodeepseek250s16346no—
django-15957djangodeepseek1,076s60806no—
django-15987djangodeepseek545s33218yes—
django-16256djangodeepseek558s411,698yes—
django-16263djangodeepseek1,646s1301,520no—
django-16454djangodeepseek449s28298no—
django-16502djangodeepseek752s28339no—
django-16631djangodeepseek718s31368no—
django-16667djangodeepseek300s19368no—
matplotlib-21568matplotlibdeepseek1,169s69263no—
matplotlib-23299matplotlibdeepseek530s26555no—
matplotlib-24870matplotlibdeepseek550s45557yes—
matplotlib-26208matplotlibdeepseek911s51621no—
matplotlib-26466matplotlibdeepseek484s40822no—
seaborn-3187seaborndeepseek730s49939no—
xarray-3993xarraydeepseek305s25318yes—
xarray-4687xarraydeepseek389s39958no—
xarray-6938xarraydeepseek347s35601no—
xarray-6992xarraydeepseek198s22274no—
xarray-7229xarraydeepseek600s45840no—
pylint-4551pylintdeepseek851s1011,105no—
pylint-4604pylintdeepseek109s13195no—
pylint-4661pylintdeepseek291s16173no—
pylint-8898pylintdeepseek454s43539no—
pytest-10356pytestdeepseek421s36562no—
sphinx-10466sphinxdeepseek305s33372yes—
sphinx-7462sphinxdeepseek437s24124no—
flash + scope cards · 67 previously-solved instances
astropy-12907astropydeepseek435s16493yes—
astropy-13236astropydeepseek666s34277no—
astropy-13453astropydeepseek480s29782yes—
astropy-13579astropydeepseek822s311,078yes—
astropy-14096astropydeepseek673s23142yes—
astropy-14309astropydeepseek389s19208yes—
astropy-14508astropydeepseek643s28232yes—
astropy-14539astropydeepseek481s24202yes—
astropy-14995astropydeepseek330s17331yes—
astropy-7166astropydeepseek597s38193yes—
astropy-7336astropydeepseek354s1265yes—
astropy-7671astropydeepseek409s15105yes—
astropy-8707astropydeepseek529s29597yes—
astropy-8872astropydeepseek501s23262yes—
django-10880djangodeepseek326s22389yes—
django-10914djangodeepseek233s15185yes—
django-11066djangodeepseek106s6134yes—
django-11095djangodeepseek169s1277yes—
django-11099djangodeepseek137s1150yes—
django-11119djangodeepseek180s1866yes—
django-11133djangodeepseek212s21315yes—
django-11138djangodeepseek593s43953yes—
django-11163djangodeepseek223s23124yes—
django-11179djangodeepseek173s975yes—
django-11206djangodeepseek464s22309yes—
django-11211djangodeepseek458s33422yes—
django-11265djangodeepseek502s421,177no—
django-11276djangodeepseek294s22153yes—
django-11292djangodeepseek340s30720yes—
django-11299djangodeepseek234s28211yes—
django-11333djangodeepseek253s18147yes—
django-11400djangodeepseek418s29371no—
django-11433djangodeepseek374s22227yes—
django-11451djangodeepseek165s13196yes—
django-11490djangodeepseek413s32648yes—
django-11551djangodeepseek243s20108yes—
django-11555djangodeepseek630s33369yes—
django-11603djangodeepseek172s13204yes—
django-11728djangodeepseek348s26203yes—
django-11740djangodeepseek378s41421yes—
django-11749djangodeepseek650s41133yes—
django-11790djangodeepseek238s20138yes—
django-11815djangodeepseek467s29163yes—
django-11880djangodeepseek170s21203yes—
django-11951djangodeepseek287s35340yes—
django-11964djangodeepseek579s23606yes—
django-11999djangodeepseek182s1661yes—
django-12039djangodeepseek291s21431yes—
django-12050djangodeepseek210s18153yes—
django-12125djangodeepseek620s31943yes—
django-12143djangodeepseek152s1119yes—
django-12155djangodeepseek164s1376yes—
django-12193djangodeepseek409s35260yes—
django-12262djangodeepseek303s32259yes—
django-12276djangodeepseek174s14294yes—
django-12304djangodeepseek512s30525yes—
django-12308djangodeepseek273s20349no—
django-12419djangodeepseek212s15408yes—
django-12708djangodeepseek406s31321yes—
django-12713djangodeepseek215s18145yes—
django-12741djangodeepseek145s8138yes—
django-12858djangodeepseek490s35493yes—
django-13012djangodeepseek385s24308yes—
django-13023djangodeepseek200s13252yes—
django-13028djangodeepseek498s26216yes—
django-13089djangodeepseek310s21262yes—
django-13109djangodeepseek248s19127yes—
worker + tester · 158 rows recorded
pylint-4604pylintflash tester195s1288no—
sphinx-9711sphinxflash tester120s1692no—
pylint-4661pylintflash tester335s33217no—
django-14140djangoflash tester463s30344no—
django-11239djangoflash tester498s31444no—
django-10999djangoflash tester548s2144no—
django-11820djangoflash tester548s23176no—
django-11532djangoflash tester590s3199no—
sphinx-7462sphinxflash tester260s1455no—
django-14315djangoflash tester630s29336yes—
django-14559djangoflash tester742s——crash—
django-14170djangoflash tester816s32461no—
sympy-16886sympyflash tester862s——crash—
django-15127djangoflash tester453s27354no—
django-12774djangoflash tester388s31360yes—
astropy-7606astropyflash tester988s——crash—
django-11477djangoflash tester4s——crash—
astropy-14598astropyflash tester682s36603no—
django-14376djangoflash tester5s——crash—
sphinx-9602sphinxflash tester33s——crash—
sympy-18199sympyflash tester33s——crash—
django-12325djangoflash tester452s36300no—
pylint-8898pylintflash tester18s——crash—
django-15916djangoflash tester18s——crash—
sympy-20916sympyflash tester21s——crash—
django-11848djangoflash tester208s12115no—
django-14155djangoflash tester30s——crash—
django-14034djangoflash tester40s——crash—
astropy-14182astropyflash tester20s——crash—
django-12406djangoflash tester21s——crash—
django-15987djangoflash tester11s——crash—
sphinx-7748sphinxflash tester3s——crash—
django-13195djangoflash tester295s21239no—
sphinx-10466sphinxflash tester21s——crash—
django-13297djangoflash tester22s——crash—
django-11141djangoflash tester20s——crash—
django-14534djangoflash tester22s——crash—
sphinx-8056sphinxflash tester3s——crash—
sphinx-7590sphinxflash tester6s——crash—
sphinx-9461sphinxflash tester6s——crash—
sympy-21930sympyflash tester5s——crash—
astropy-7606astropysonnet tester1s——crash—
django-14559djangosonnet tester3s——crash—
django-14011djangosonnet tester3s——crash—
django-13794djangosonnet tester3s——crash—
django-11149djangosonnet tester3s——crash—
django-14376djangosonnet tester4s——crash—
django-13513djangosonnet tester4s——crash—
xarray-3993xarraysonnet tester4s——crash—
django-11477djangosonnet tester6s——crash—
sympy-18763sympysonnet tester6s——crash—
sphinx-9602sphinxsonnet tester5s——crash—
django-16667djangosonnet tester7s——crash—
sympy-16886sympysonnet tester8s——crash—
django-15916djangosonnet tester5s——crash—
sympy-18199sympysonnet tester5s——crash—
pylint-8898pylintsonnet tester6s——crash—
sympy-20916sympysonnet tester7s——crash—
django-14034djangosonnet tester7s——crash—
django-14155djangosonnet tester7s——crash—
astropy-14182astropysonnet tester5s——crash—
django-12406djangosonnet tester7s——crash—
sphinx-7748sphinxsonnet tester6s——crash—
django-15987djangosonnet tester7s——crash—
sphinx-10466sphinxsonnet tester6s——crash—
django-13297djangosonnet tester7s——crash—
django-14534djangosonnet tester7s——crash—
django-11141djangosonnet tester12s——crash—
sympy-16886sympysonnet tester1s——crash—
django-16667djangosonnet tester3s——crash—
django-14011djangosonnet tester3s——crash—
django-14376djangosonnet tester3s——crash—
django-13794djangosonnet tester4s——crash—
django-14559djangosonnet tester4s——crash—
django-11149djangosonnet tester4s——crash—
astropy-7606astropysonnet tester5s——crash—
xarray-3993xarraysonnet tester5s——crash—
sympy-18763sympysonnet tester5s——crash—
django-11477djangosonnet tester6s——crash—
sympy-18199sympysonnet tester5s——crash—
sphinx-9602sphinxsonnet tester8s——crash—
pylint-8898pylintsonnet tester6s——crash—
django-13513djangosonnet tester11s——crash—
django-15916djangosonnet tester7s——crash—
sympy-20916sympysonnet tester7s——crash—
django-14034djangosonnet tester7s——crash—
django-14155djangosonnet tester7s——crash—
astropy-14182astropysonnet tester6s——crash—
sphinx-7748sphinxsonnet tester5s——crash—
django-12406djangosonnet tester7s——crash—
django-15987djangosonnet tester7s——crash—
sphinx-10466sphinxsonnet tester5s——crash—
django-13297djangosonnet tester6s——crash—
django-14534djangosonnet tester18s——crash—
sphinx-9461sphinxsonnet tester42s——crash—
django-11141djangosonnet tester44s——crash—
sympy-15875sympysonnet tester43s——crash—
django-16454djangosonnet tester42s——crash—
sympy-21930sympysonnet tester42s——crash—
astropy-13033astropysonnet tester43s——crash—
sphinx-7590sphinxsonnet tester43s——crash—
sphinx-8056sphinxsonnet tester43s——crash—
django-12273djangosonnet tester42s——crash—
django-15695djangosonnet tester40s——crash—
xarray-4687xarraysonnet tester28s——crash—
xarray-6938xarraysonnet tester104s——crash—
django-10973djangosonnet tester92s——crash—
xarray-6992xarraysonnet tester66s——crash—
django-14792djangosonnet tester70s——crash—
django-16502djangosonnet tester70s——crash—
django-15252djangosonnet tester70s——crash—
django-13344djangosonnet tester70s——crash—
django-13212djangosonnet tester70s——crash—
django-15037djangosonnet tester70s——crash—
sympy-22080sympysonnet tester42s——crash—
django-16631djangosonnet tester70s——crash—
pytest-10356pytestsonnet tester70s——crash—
django-11477djangosonnet tester294s1990no—
django-14376djangosonnet tester346s24246no—
xarray-3993xarraysonnet tester420s37166no—
django-14559djangosonnet tester440s——crash—
django-16667djangosonnet tester475s22212no—
astropy-7606astropysonnet tester530s1983yes—
django-13513djangosonnet tester556s3264——
django-11149djangosonnet tester581s43540yes—
django-13794djangosonnet tester703s42395no—
sympy-18763sympysonnet tester719s1825no—
sympy-16886sympysonnet tester729s1736yes—
django-15916djangosonnet tester396s27319no—
django-14155djangosonnet tester418s23136no—
sphinx-9602sphinxsonnet tester750s59628no—
django-14034djangosonnet tester637s49478no—
django-14011djangosonnet tester1,225s52538no—
django-12406djangosonnet tester683s38565no—
pylint-8898pylintsonnet tester832s47781no—
django-15987djangosonnet tester566s——crash—
sympy-18199sympysonnet tester956s36170no—
sphinx-10466sphinxsonnet tester402s52296yes—
django-13297djangosonnet tester683s35318no—
sympy-20916sympysonnet tester1,095s40352no—
astropy-14182astropysonnet tester1,094s——crash—
sphinx-8056sphinxsonnet tester683s43958yes—
django-14534djangosonnet tester964s53444no—
sphinx-9461sphinxsonnet tester719s48798no—
sphinx-7590sphinxsonnet tester759s53589no—
django-12273djangosonnet tester629s34236no—
django-16454djangosonnet tester620s32340no—
django-11141djangosonnet tester1,109s——crash—
django-10973djangosonnet tester313s21248no—
django-15695djangosonnet tester644s41575no—
sympy-21930sympysonnet tester1,126s34135no—
django-13212djangosonnet tester552s27645no—
pytest-10356pytestsonnet tester770s59331no—
xarray-6992xarraysonnet tester884s——crash—
sympy-15875sympysonnet tester1,657s83255no—
xarray-4687xarraysonnet tester936s75587yes—
astropy-13033astropysonnet tester1,717s——crash—
django-16631djangosonnet tester612s30344no—
worker + verifier · 43 rows recorded
django-14559djangoflash verifier196s959yes—
django-11239djangoflash verifier200s11138no—
django-14315djangoflash verifier283s19333no—
astropy-7606astropyflash verifier309s1480yes—
django-10999djangoflash verifier330s1791no—
pylint-4604pylintflash verifier344s33202no—
sphinx-9711sphinxflash verifier153s18160no—
django-14140djangoflash verifier365s2446no—
pylint-4661pylintflash verifier371s29446no—
django-14170djangoflash verifier416s22476no—
django-11820djangoflash verifier430s21262no—
sphinx-7462sphinxflash verifier201s2090no—
django-15127djangoflash verifier202s12246no—
sympy-16886sympyflash verifier530s1057yes—
django-13513djangoflash verifier221s25172——
xarray-3993xarrayflash verifier264s23228yes—
django-11532djangoflash verifier608s23269no—
django-13195djangoflash verifier222s20130yes—
django-12325djangoflash verifier337s20155no—
django-12774djangoflash verifier372s30328no—
django-16667djangoflash verifier200s1449no—
django-14376djangoflash verifier229s19310no—
django-11477djangoflash verifier267s2079no—
django-13794djangoflash verifier453s32168no—
sphinx-9602sphinxflash verifier309s33691no—
django-11848djangoflash verifier604s29202no—
astropy-14598astropyflash verifier1,041s47989no—
sympy-18199sympyflash verifier643s18184no—
django-15916djangoflash verifier644s36467yes—
django-11149djangoflash verifier935s511,094yes—
django-14155djangoflash verifier516s26341no—
django-14034djangoflash verifier534s21488no—
django-13513djangoflash verifier162s1687——
sphinx-10466sphinxsonnet verifier300s34546yes—
django-11141djangosonnet verifier179s13169no—
django-14534djangosonnet verifier394s30167no—
django-15987djangosonnet verifier428s26250no—
astropy-14182astropysonnet verifier432s29362no—
django-13297djangosonnet verifier513s28473——
django-12406djangosonnet verifier633s25485——
django-14011djangosonnet verifier678s28407——
sphinx-8056sphinxsonnet verifier495s56470——
sympy-18763sympysonnet verifier880s37341——
worker + symptom map · 53 rows recorded
pylint-4604pylintsymptom map152s2695no—
django-14559djangosymptom map153s11312no—
django-11239djangosymptom map179s17217yes—
django-14315djangosymptom map212s15328yes—
django-14140djangosymptom map216s16145no—
pylint-4661pylintsymptom map241s30225no—
sphinx-9711sphinxsymptom map111s1386no—
django-10999djangosymptom map264s13149no—
django-14170djangosymptom map268s23372no—
astropy-7606astropysymptom map280s12125yes—
sphinx-7462sphinxsymptom map107s16132no—
django-11532djangosymptom map393s33312no—
django-11820djangosymptom map400s20273no—
django-15127djangosymptom map212s12244no—
xarray-3993xarraysymptom map257s28281no—
django-12774djangosymptom map250s1364yes—
sympy-16886sympysymptom map511s936yes—
django-12325djangosymptom map263s20198no—
django-13513djangosymptom map286s1589——
django-13195djangosymptom map194s14172no—
django-16667djangosymptom map176s1278no—
django-13794djangosymptom map384s24304yes—
django-11848djangosymptom map290s13135no—
django-11477djangosymptom map256s27317no—
sphinx-9602sphinxsymptom map229s27211no—
astropy-14598astropysymptom map691s37530no—
django-15916djangosymptom map322s22295no—
django-14376djangosymptom map415s30416no—
django-14034djangosymptom map319s23555no—
django-14155djangosymptom map274s20220no—
django-11149djangosymptom map746s511,354yes—
pylint-8898pylintsymptom map495s52516no—
django-15987djangosymptom map205s16282no—
sympy-18763sympysymptom map646s1832yes—
sympy-18199sympysymptom map589s21353no—
astropy-14182astropysymptom map376s21380no—
django-12406djangosymptom map481s32640no—
django-14011djangosymptom map980s44489no—
sphinx-10466sphinxsymptom map256s34577yes—
django-14534djangosymptom map327s26483no—
sphinx-7590sphinxsymptom map296s28369no—
django-13297djangosymptom map408s14316no—
django-11141djangosymptom map409s42321no—
sphinx-7748sphinxsymptom map614s591,275no—
sympy-20916sympysymptom map904s23306——
astropy-13033astropysymptom map420s22245——
django-10973djangosymptom map153s11187——
xarray-6938xarraysymptom map249s31334——
sphinx-8056sphinxsymptom map554s53561——
django-16454djangosymptom map325s19112——
django-12273djangosymptom map432s22137——
sympy-15875sympysymptom map611s1338——
xarray-6992xarraysymptom map199s18357——
run 2 · instances run first (the 114 previously missed) · 28 resolved
django-11239djangoflash stack239s17243no—
django-14315djangoflash stack260s18406no—
django-14559djangoflash stack274s15158yes—
django-14140djangoflash stack301s1635no—
astropy-7606astropyflash stack316s17106yes—
pylint-4661pylintflash stack335s30214no—
pylint-4604pylintflash stack362s26208no—
django-11820djangoflash stack388s26405no—
sphinx-9711sphinxflash stack230s26162yes—
django-10999djangoflash stack562s34266no—
xarray-3993xarrayflash stack224s27324yes—
sphinx-7462sphinxflash stack302s22200no—
django-14170djangoflash stack620s35537yes—
django-11532djangoflash stack421s22226no—
django-12774djangoflash stack295s1873yes—
django-15127djangoflash stack373s27372no—
django-13513djangoflash stack186s1491no—
sympy-16886sympyflash stack570s17120no—
django-13195djangoflash stack304s15268no—
django-16667djangoflash stack292s1880no—
django-13794djangoflash stack384s24228no—
django-11149djangoflash stack542s33641yes—
astropy-14598astropyflash stack865s41642no—
django-12325djangoflash stack601s25230no—
django-11848djangoflash stack525s23239no—
django-14376djangoflash stack258s20350no—
sphinx-9602sphinxflash stack302s26188no—
django-14011djangoflash stack948s42721no—
sympy-18763sympyflash stack712s25283no—
django-14155djangoflash stack243s14160no—
pylint-8898pylintflash stack532s48747no—
sympy-18199sympyflash stack757s27304no—
django-15916djangoflash stack669s35454yes—
django-11477djangoflash stack1,059s78730no—
django-15987djangoflash stack492s22243yes—
django-14034djangoflash stack780s32677no—
sphinx-7748sphinxflash stack540s531,221no—
astropy-14182astropyflash stack605s39631no—
django-12406djangoflash stack784s34534yes—
django-11141djangoflash stack358s20474no—
sphinx-10466sphinxflash stack624s761,158yes—
django-13297djangoflash stack823s30327yes—
django-14534djangoflash stack712s39460no—
sphinx-8056sphinxflash stack634s411,092yes—
sympy-20916sympyflash stack1,541s58321no—
astropy-13033astropyflash stack672s27394no—
sphinx-9461sphinxflash stack672s401,047no—
sphinx-7590sphinxflash stack786s62725no—
sympy-15875sympyflash stack886s27167yes—
django-12273djangoflash stack585s31174no—
django-16454djangoflash stack484s24330yes—
django-10973djangoflash stack359s22269no—
xarray-6938xarrayflash stack518s32461no—
django-15695djangoflash stack576s29242no—
xarray-6992xarrayflash stack367s25540no—
sympy-21930sympyflash stack980s34250no—
django-13212djangoflash stack575s32684no—
django-15037djangoflash stack408s30877yes—
django-16631djangoflash stack511s17404no—
xarray-4687xarrayflash stack1,044s106656yes—
django-16502djangoflash stack690s27439no—
django-14792djangoflash stack804s35866no—
django-15252djangoflash stack861s382,448no—
pytest-10356pytestflash stack1,090s71801no—
django-11885djangoflash stack705s31997no—
astropy-14365astropyflash stack406s23304yes—
seaborn-3187seabornflash stack513s——crash—
django-13344djangoflash stack1,257s511,367no—
sphinx-10614sphinxflash stack653s52457no—
matplotlib-14623matplotlibflash stack715s36615yes—
sympy-22080sympyflash stack1,369s411,124no—
sympy-20428sympyflash stack1,042s38497no—
xarray-7229xarrayflash stack690s50989no—
sympy-20438sympyflash stack1,393s641,072no—
pylint-4551pylintflash stack683s651,426no—
matplotlib-26466matplotlibflash stack905s40768no—
pylint-7080pylintflash stack639s48536no—
sympy-21596sympyflash stack1,053s27462no—
django-10097djangoflash stack380s26137yes—
astropy-14369astropyflash stack813s39928no—
django-15563djangoflash stack1,111s49553yes—
pylint-4970pylintflash stack743s911,454no—
django-15629djangoflash stack901s64964no—
django-16256djangoflash stack610s31903no—
astropy-13977astropyflash stack935s47463no—
sphinx-11510sphinxflash stack792s541,045no—
django-15098djangoflash stack1,208s56364no—
django-13406djangoflash stack1,118s47743no—
django-15957djangoflash stack737s41575no—
django-11734djangoflash stack1,249s67735no—
matplotlib-25479matplotlibflash stack751s41414no—
astropy-13398astropyflash stack787s37652no—
django-11087djangoflash stack1,340s431,224no—
matplotlib-24870matplotlibflash stack875s53838no—
django-15973djangoflash stack1,482s671,302no—
matplotlib-23476matplotlibflash stack993s40450no—
matplotlib-26208matplotlibflash stack1,044s72977no—
sympy-23413sympyflash stack1,765s37564yes—
matplotlib-23299matplotlibflash stack774s45536no—
django-10554djangoflash stack919s49869no—
sympy-13798sympyflash stack2,364s23275no—
sphinx-8638sphinxflash stack674s601,799no—
sympy-14711sympyflash stack2,340s22475yes—
sympy-13974sympyflash stack3,019s51787no—
django-16263djangoflash stack1,764s911,245no—
matplotlib-21568matplotlibflash stack890s43159no—
sphinx-8548sphinxflash stack1,914s1371,135no—
sympy-13852sympyflash stack2,607s32343no—
scikit-learn-14710scikit-learnflash stack1,105s27751yes—
sympy-16597sympyflash stack7,034s1391,554no—
scikit-learn-26194scikit-learnflash stack1,835s25484yes—
sympy-14248sympyflash stack3,062s40459yes—
scikit-learn-25747scikit-learnflash stack2,554s61648no—
sphinx-9229sphinxflash stack7,431s——crash—
sphinx-9229sphinxflash stack · retry890s851,249yes—
seaborn-3187seabornflash stack · retry1,065s651,518yes—
run 2 · remaining 386 instances · 362 resolved
pytest-5809pytestflash stack174s11176yes—
requests-1766requestsflash stack241s15224yes—
xarray-4629xarrayflash stack253s17164yes—
requests-1724requestsflash stack262s18228yes—
pytest-7982pytestflash stack285s17131yes—
requests-2317requestsflash stack319s22204yes—
pytest-7521pytestflash stack400s26304yes—
requests-2931requestsflash stack402s24268yes—
django-12741djangoflash stack161s962yes—
flask-5014flaskflash stack167s22407yes—
sphinx-9367sphinxflash stack234s16323yes—
django-12143djangoflash stack179s1323yes—
requests-1921requestsflash stack585s26138yes—
requests-5414requestsflash stack586s36499yes—
pytest-10081pytestflash stack416s21651yes—
pytest-6202pytestflash stack314s25205yes—
pytest-7571pytestflash stack224s25471yes—
django-14089djangoflash stack160s10107yes—
pytest-5631pytestflash stack478s18248yes—
sphinx-9698sphinxflash stack210s22223yes—
django-13670djangoflash stack231s17183yes—
pylint-7277pylintflash stack421s19291yes—
django-11451djangoflash stack213s15259yes—
xarray-4094xarrayflash stack413s30307yes—
requests-1142requestsflash stack273s32252yes—
django-11163djangoflash stack259s25258yes—
pytest-5262pytestflash stack184s18117yes—
django-9296djangoflash stack145s1332yes—
sympy-24539sympyflash stack474s27229yes—
xarray-4075xarrayflash stack124s10161yes—
sphinx-9281sphinxflash stack220s19125yes—
pytest-8399pytestflash stack259s30202yes—
django-12050djangoflash stack253s16145yes—
django-11066djangoflash stack234s18230yes—
sphinx-9320sphinxflash stack253s19184yes—
django-12276djangoflash stack322s23301yes—
sphinx-8269sphinxflash stack222s17592yes—
pytest-7432pytestflash stack269s17151yes—
django-13410djangoflash stack202s10189yes—
django-13741djangoflash stack300s23284yes—
django-13023djangoflash stack326s20233yes—
django-13658djangoflash stack389s28214yes—
django-13028djangoflash stack538s1720yes—
django-13551djangoflash stack289s18332yes—
sympy-24562sympyflash stack549s29276yes—
django-11179djangoflash stack459s34302yes—
django-13820djangoflash stack277s13121yes—
django-14787djangoflash stack444s13537yes—
django-16527djangoflash stack264s13203yes—
sympy-24661sympyflash stack567s25504yes—
django-14373djangoflash stack202s1486yes—
django-16801djangoflash stack246s16142yes—
django-16139djangoflash stack287s15106yes—
django-13516djangoflash stack330s21112yes—
django-16493djangoflash stack321s20245yes—
django-11299djangoflash stack425s24392yes—
django-14765djangoflash stack282s17108yes—
django-14915djangoflash stack310s2289yes—
django-16612djangoflash stack463s37289yes—
django-15467djangoflash stack403s23245yes—
django-15104djangoflash stack410s30180yes—
sphinx-8475sphinxflash stack232s21418yes—
django-16429djangoflash stack285s14118yes—
django-13089djangoflash stack405s23189yes—
django-11880djangoflash stack329s24288yes—
django-13786djangoflash stack235s20500yes—
django-13279djangoflash stack644s30460yes—
django-16255djangoflash stack246s13126yes—
django-11951djangoflash stack208s15155yes—
pytest-7236pytestflash stack516s27759yes—
django-11095djangoflash stack334s20226yes—
django-11099djangoflash stack191s1428yes—
sphinx-9591sphinxflash stack492s38375yes—
django-13109djangoflash stack303s1721yes—
django-14238djangoflash stack251s17112yes—
django-15103djangoflash stack290s26139yes—
django-16642djangoflash stack414s22142yes—
django-15741djangoflash stack354s2885yes—
django-12262djangoflash stack414s29180yes—
pytest-7490pytestflash stack486s31176yes—
django-11333djangoflash stack384s28356yes—
django-11119djangoflash stack206s20472yes—
django-14752djangoflash stack222s15136yes—
django-14580djangoflash stack431s23596yes—
django-13363djangoflash stack339s23395yes—
django-14434djangoflash stack491s43369yes—
django-15851djangoflash stack273s17462yes—
django-11551djangoflash stack259s1896yes—
django-14608djangoflash stack383s26461yes—
sympy-24443sympyflash stack766s271,020yes—
xarray-3677xarrayflash stack208s20497yes—
sphinx-7757sphinxflash stack352s27391yes—
django-15380djangoflash stack392s32533yes—
sphinx-7889sphinxflash stack951s58608yes—
sympy-19954sympyflash stack639s14110yes—
django-16100djangoflash stack247s14255yes—
django-13449djangoflash stack627s20358yes—
django-13837djangoflash stack581s24211yes—
sympy-16766sympyflash stack782s35358yes—
django-11740djangoflash stack682s37460yes—
django-12304djangoflash stack346s21122yes—
django-10914djangoflash stack252s19251yes—
requests-6028requestsflash stack720s53409yes—
django-15277djangoflash stack279s14206yes—
sphinx-8551sphinxflash stack548s41936yes—
sympy-19637sympyflash stack854s27180yes—
sympy-22914sympyflash stack807s31489yes—
pytest-7205pytestflash stack181s21183yes—
xarray-4966xarrayflash stack316s18337yes—
sympy-15809sympyflash stack649s17328yes—
sympy-15349sympyflash stack737s18128yes—
django-12155djangoflash stack232s17151yes—
django-13343djangoflash stack400s20114yes—
sphinx-9673sphinxflash stack452s41465yes—
pylint-6903pylintflash stack324s28233yes—
django-11433djangoflash stack546s31509yes—
sympy-17139sympyflash stack809s17168yes—
django-12308djangoflash stack563s39451yes—
pytest-10051pytestflash stack264s18248yes—
django-13964djangoflash stack1,228s53753yes—
django-11603djangoflash stack221s16203yes—
sympy-23824sympyflash stack920s19402yes—
django-14311djangoflash stack450s29195no—
sympy-23534sympyflash stack766s21329yes—
django-13810djangoflash stack284s16379yes—
django-16145djangoflash stack420s22413yes—
django-15499djangoflash stack472s34622yes—
sympy-23262sympyflash stack802s33329yes—
django-15278djangoflash stack401s20298yes—
django-12039djangoflash stack437s30939yes—
sphinx-8721sphinxflash stack490s34418yes—
scikit-learn-9288scikit-learnflash stack713s17472yes—
django-13568djangoflash stack455s23348yes—
django-14493djangoflash stack576s33523yes—
django-16662djangoflash stack322s19308yes—
sphinx-8459sphinxflash stack214s221,081yes—
django-12708djangoflash stack396s34300yes—
sympy-15345sympyflash stack689s25576yes—
sphinx-8595sphinxflash stack380s38265yes—
sympy-24213sympyflash stack1,057s37309yes—
django-11728djangoflash stack503s20177yes—
django-12125djangoflash stack520s27420yes—
sympy-18189sympyflash stack755s19270yes—
xarray-3305xarrayflash stack151s19352yes—
django-12193djangoflash stack402s28189yes—
sympy-20590sympyflash stack937s2560yes—
django-13821djangoflash stack736s25813yes—
sphinx-8265sphinxflash stack692s42730yes—
django-16082djangoflash stack359s32740yes—
django-11555djangoflash stack733s47305yes—
django-15569djangoflash stack302s25167yes—
sphinx-9658sphinxflash stack508s43614yes—
sympy-15017sympyflash stack1,371s60966yes—
django-13012djangoflash stack440s28623yes—
django-16877djangoflash stack377s23291yes—
django-13569djangoflash stack303s21286yes—
django-11749djangoflash stack346s16118yes—
django-11999djangoflash stack365s25244yes—
django-14771djangoflash stack303s14122yes—
django-13590djangoflash stack512s35121yes—
sphinx-8593sphinxflash stack539s61882yes—
xarray-3151xarrayflash stack291s301,540yes—
django-16333djangoflash stack480s31221yes—
django-15382djangoflash stack752s40677yes—
django-15731djangoflash stack356s23222yes—
django-15315djangoflash stack216s1740yes—
django-10880djangoflash stack363s32106yes—
django-12713djangoflash stack252s21192yes—
django-15368djangoflash stack320s2766yes—
django-16595djangoflash stack289s22394yes—
django-11490djangoflash stack654s34683yes—
django-11815djangoflash stack413s29534yes—
django-11211djangoflash stack535s27334yes—
sympy-21847sympyflash stack750s21194yes—
django-17029djangoflash stack239s1652yes—
django-15863djangoflash stack673s29217yes—
sympy-19346sympyflash stack967s37218yes—
django-16136djangoflash stack563s20741no—
sphinx-9258sphinxflash stack944s631,040yes—
django-13033djangoflash stack755s43583yes—
django-14122djangoflash stack296s23382yes—
django-13417djangoflash stack769s38230yes—
django-15268djangoflash stack560s28547yes—
django-13315djangoflash stack542s34449yes—
django-14007djangoflash stack615s28250yes—
sympy-20154sympyflash stack966s26530yes—
django-11292djangoflash stack345s18714yes—
pylint-6528pylintflash stack402s34441yes—
sympy-19495sympyflash stack1,403s39301yes—
sympy-16450sympyflash stack955s31416yes—
pytest-5787pytestflash stack498s30905yes—
sphinx-7910sphinxflash stack480s34437yes—
django-16899djangoflash stack258s1473yes—
django-11276djangoflash stack433s24363yes—
django-17087djangoflash stack390s28125yes—
django-16116djangoflash stack437s24753yes—
django-11964djangoflash stack627s27547yes—
django-13512djangoflash stack358s22914yes—
django-13346djangoflash stack692s28796yes—
matplotlib-13989matplotlibflash stack223s15339yes—
django-14017djangoflash stack683s38456yes—
django-13807djangoflash stack303s19111yes—
django-15572djangoflash stack412s21668yes—
sphinx-11445sphinxflash stack302s31306yes—
django-13401djangoflash stack596s32194no—
xarray-3095xarrayflash stack180s24577yes—
django-11400djangoflash stack593s34250no—
sympy-19783sympyflash stack1,046s31444yes—
django-15375djangoflash stack541s31263yes—
django-15930djangoflash stack560s23423yes—
django-13925djangoflash stack520s42268yes—
sphinx-7454sphinxflash stack385s261,304no—
sympy-17318sympyflash stack1,363s36478yes—
django-15561djangoflash stack379s22306yes—
django-16901djangoflash stack368s28341yes—
sympy-20801sympyflash stack1,171s29266yes—
astropy-14539astropyflash stack503s26577yes—
django-7530djangoflash stack372s21204yes—
django-14539djangoflash stack368s17154yes—
astropy-7166astropyflash stack374s16130yes—
sphinx-9230sphinxflash stack289s29155yes—
django-16485djangoflash stack363s19146yes—
django-16315djangoflash stack534s41437yes—
sphinx-8035sphinxflash stack444s43486yes—
django-11206djangoflash stack1,128s51237yes—
django-14631djangoflash stack522s34576yes—
xarray-4356xarrayflash stack410s30212yes—
sympy-15976sympyflash stack1,022s28245yes—
astropy-14995astropyflash stack454s18281yes—
astropy-7336astropyflash stack325s17199yes—
django-15814djangoflash stack500s22346yes—
sympy-16792sympyflash stack914s431,009yes—
pylint-6386pylintflash stack434s40860yes—
django-12965djangoflash stack762s39816yes—
xarray-7233xarrayflash stack393s28456yes—
xarray-6721xarrayflash stack396s31187yes—
django-15525djangoflash stack409s27824yes—
django-13809djangoflash stack424s21721yes—
django-12754djangoflash stack809s62896yes—
django-14500djangoflash stack698s31994yes—
django-13158djangoflash stack774s62574yes—
django-16819djangoflash stack455s28542yes—
sympy-22714sympyflash stack913s32658yes—
django-14999djangoflash stack528s44830yes—
django-13112djangoflash stack502s441,166yes—
sympy-11618sympyflash stack284s2178yes—
sphinx-8120sphinxflash stack363s34958yes—
astropy-12907astropyflash stack458s21377yes—
django-12419djangoflash stack385s20371yes—
django-13121djangoflash stack761s31577yes—
sympy-12096sympyflash stack518s20201yes—
sphinx-8621sphinxflash stack333s15210yes—
django-14855djangoflash stack474s37692yes—
xarray-6744xarrayflash stack449s281,376yes—
django-14053djangoflash stack485s30381no—
sphinx-10435sphinxflash stack483s41146no—
sphinx-10323sphinxflash stack275s36591yes—
xarray-6461xarrayflash stack289s29516no—
matplotlib-24149matplotlibflash stack670s28276yes—
django-15732djangoflash stack625s36336yes—
astropy-7671astropyflash stack560s2199yes—
django-14351djangoflash stack1,008s44686yes—
astropy-14309astropyflash stack467s22257yes—
sympy-18211sympyflash stack959s27379yes—
django-11790djangoflash stack275s2384no—
sympy-24066sympyflash stack1,915s501,013yes—
django-14404djangoflash stack484s2136yes—
sphinx-10449sphinxflash stack544s40933yes—
sympy-23950sympyflash stack1,788s52367yes—
astropy-13236astropyflash stack784s32428no—
sympy-21612sympyflash stack1,464s46734yes—
django-15280djangoflash stack978s48979yes—
astropy-13453astropyflash stack581s24755yes—
django-15128djangoflash stack892s50346yes—
sympy-13031sympyflash stack722s53438yes—
django-11265djangoflash stack1,401s881,244yes—
django-12209djangoflash stack1,313s48273yes—
django-16938djangoflash stack601s26389no—
sphinx-10673sphinxflash stack796s67541yes—
sympy-12481sympyflash stack451s25581yes—
astropy-14508astropyflash stack591s29215yes—
django-15161djangoflash stack731s43422yes—
sympy-15599sympyflash stack1,826s53489yes—
sphinx-7985sphinxflash stack932s51781yes—
django-15022djangoflash stack1,009s49697yes—
xarray-6599xarrayflash stack721s47611no—
sympy-17655sympyflash stack1,991s62492yes—
matplotlib-24570matplotlibflash stack456s24272yes—
django-12858djangoflash stack837s44882yes—
scikit-learn-26323scikit-learnflash stack1,799s22425yes—
sphinx-7440sphinxflash stack986s631,016no—
sympy-17630sympyflash stack1,236s541,230yes—
scikit-learn-13124scikit-learnflash stack614s19629yes—
django-13128djangoflash stack805s31988yes—
scikit-learn-13439scikit-learnflash stack605s21150yes—
astropy-8872astropyflash stack774s37385yes—
django-16032djangoflash stack1,021s56621yes—
pytest-5840pytestflash stack1,211s67367no—
scikit-learn-13142scikit-learnflash stack520s12162yes—
sympy-21379sympyflash stack1,365s39509yes—
matplotlib-24637matplotlibflash stack441s23229yes—
xarray-2905xarrayflash stack409s23540yes—
django-15503djangoflash stack655s33422yes—
astropy-8707astropyflash stack592s32515yes—
django-17084djangoflash stack683s28856yes—
scikit-learn-12973scikit-learnflash stack498s17344yes—
matplotlib-26342matplotlibflash stack439s19193yes—
django-16950djangoflash stack1,126s57749no—
astropy-13579astropyflash stack1,188s521,135yes—
scikit-learn-13328scikit-learnflash stack615s17595yes—
scikit-learn-14053scikit-learnflash stack691s19265yes—
matplotlib-24177matplotlibflash stack706s31431yes—
seaborn-3069seabornflash stack1,152s491,185yes—
sympy-18698sympyflash stack2,012s67442yes—
scikit-learn-10908scikit-learnflash stack594s27172yes—
scikit-learn-12585scikit-learnflash stack623s28135yes—
scikit-learn-14141scikit-learnflash stack589s11207yes—
django-16560djangoflash stack746s35616yes—
scikit-learn-10844scikit-learnflash stack545s16244yes—
django-16661djangoflash stack723s26218no—
scikit-learn-15100scikit-learnflash stack600s823yes—
astropy-14096astropyflash stack654s20124yes—
scikit-learn-10297scikit-learnflash stack566s21320yes—
django-14725djangoflash stack870s39916no—
scikit-learn-11578scikit-learnflash stack760s26392yes—
scikit-learn-13779scikit-learnflash stack661s14144yes—
scikit-learn-25232scikit-learnflash stack2,502s381,820yes—
pytest-7324pytestflash stack583s28519no—
scikit-learn-13496scikit-learnflash stack695s26477yes—
matplotlib-20859matplotlibflash stack390s17462no—
scikit-learn-14894scikit-learnflash stack731s19112yes—
sympy-12489sympyflash stack453s20251no—
matplotlib-20676matplotlibflash stack780s39623no—
scikit-learn-14496scikit-learnflash stack737s20554yes—
matplotlib-26113matplotlibflash stack1,041s47308yes—
scikit-learn-11310scikit-learnflash stack564s20364yes—
matplotlib-22719matplotlibflash stack561s28372yes—
scikit-learn-13135scikit-learnflash stack573s15166yes—
sympy-22456sympyflash stack1,963s78808yes—
django-11138djangoflash stack952s621,252yes—
django-12663djangoflash stack2,373s92802yes—
matplotlib-24026matplotlibflash stack530s22206yes—
sympy-12419sympyflash stack796s42613yes—
sympy-19040sympyflash stack1,635s71759yes—
matplotlib-26291matplotlibflash stack841s52746yes—
matplotlib-23314matplotlibflash stack421s25143yes—
matplotlib-20488matplotlibflash stack1,194s68741yes—
matplotlib-25287matplotlibflash stack440s18350yes—
scikit-learn-14983scikit-learnflash stack921s42274yes—
matplotlib-23412matplotlibflash stack479s26480yes—
scikit-learn-14087scikit-learnflash stack972s25360yes—
matplotlib-24970matplotlibflash stack549s24205yes—
matplotlib-24627matplotlibflash stack967s54591yes—
matplotlib-25775matplotlibflash stack916s571,110yes—
django-15554djangoflash stack1,951s881,118yes—
matplotlib-25122matplotlibflash stack530s20449yes—
pytest-6197pytestflash stack1,402s87501yes—
matplotlib-22865matplotlibflash stack356s19284yes—
matplotlib-25332matplotlibflash stack506s33264yes—
sympy-13877sympyflash stack2,595s37317yes—
matplotlib-25311matplotlibflash stack828s35596yes—
matplotlib-25960matplotlibflash stack1,195s631,168yes—
matplotlib-22871matplotlibflash stack507s22562yes—
scikit-learn-12682scikit-learnflash stack1,077s24902no—
sympy-13551sympyflash stack2,711s30111yes—
sympy-14976sympyflash stack2,643s26683yes—
sympy-13647sympyflash stack2,647s3895yes—
sympy-14531sympyflash stack2,986s59381yes—
sympy-13091sympyflash stack1,010s57513yes—
scikit-learn-25973scikit-learnflash stack1,938s17262yes—
scikit-learn-25931scikit-learnflash stack2,077s36474yes—
sympy-13480sympyflash stack2,206s17192yes—
scikit-learn-25102scikit-learnflash stack2,682s49864yes—
sympy-13372sympyflash stack2,533s24273yes—
sympy-13878sympyflash stack3,496s671,837yes—
sympy-13615sympyflash stack3,327s36274yes—
sympy-13757sympyflash stack4,336s83418yes—
xarray-4695xarrayflash stack22s——crash—
xarray-7393xarrayflash stack16s——crash—
django-11133djangoflash stack3s——crash—
django-14349djangoflash stack5s——crash—
django-16569djangoflash stack3s——crash—
django-13933djangoflash stack4s——crash—
django-14672djangoflash stack390s——crash—
matplotlib-20826matplotlibflash stack36s——crash—
scikit-learn-14629scikit-learnflash stack2s——crash—
django-14672djangoflash stack · retry337s26384yes—
xarray-7393xarrayflash stack · retry361s26449yes—
django-13933djangoflash stack · retry384s11193yes—
django-16569djangoflash stack · retry391s23156yes—
django-11133djangoflash stack · retry460s17554yes—
django-14349djangoflash stack · retry473s15211yes—
xarray-4695xarrayflash stack · retry499s51474yes—
scikit-learn-14629scikit-learnflash stack · retry886s24404no—
matplotlib-20826matplotlibflash stack · retry959s35908yes—