Momentic compares AI testers against 1,396 known bugs Momentic compares AI testers against 1,396 known bugs

Coverage and cost pull in different directions

Momentic reports that Mo matched 19.3% of labeled bugs in its 60-minute condition, against 6.2% for Codex with GPT-6.1 Sol and Playwright MCP. Cost per match was $2.67 for Mo and $0.80 for that Codex configuration. Time was allocated separately to each product area. Momentic builds Mo and ran the evaluation. Candidate labels came from agent runs, followed by reproduction, independent checks and human review. A model judge matches submitted reports to those labels. Equal time budgets did not cap spending. Individual runs and dispersion were not published, limiting comparisons of small differences.

The published numbers have a version history

A November 2 repository change published a second set of results while retaining older exports. Its accompanying change note says the rerun used new rows for every configuration except Devin, whose original results remained. It also removed an earlier denominator of 1,346 bugs across nineteen areas. The updated table covers all twenty areas and the same 1,396 reference labels. Most configurations have three runs per area and budget; Devin has one. The change also revised the data-card recall figures, making the version attached to a comparison consequential.

The bill depends on what the harness counts

The repository accounts for Codex using token consumption at list prices plus estimated Browserbase time. Claude Code uses the cost reported by its agent SDK plus the browser estimate. Mo uses its session spend, while Devin’s costs were calculated separately. Sandbox infrastructure from E2B and grading are excluded. Execution arrangements also vary. Playwright MCP gives the parent and each subagent their own browser, while agent-browser shares one browser within a run. Choosing a local browser changes the setup again: Playwright runs then have no subagents and receive separate labels in merged results. Mo manages its own subagents, with a configurable cap; Devin runs without child sessions.

What the judge actually matches

The grading contract defines the judge’s job as matching reports to references, not independently reproducing a defect. It withholds source-cause and review-evidence fields from the matching inputs. A report can match when it describes the same observable failure under compatible conditions, without repeating every reference reproduction step. Conversely, different interface failures remain separate even when they share a root cause. A finding absent from the reference set cannot be rejected for that reason alone. Conflicts between positive and negative references are left unverified. Severity is excluded as matching evidence. These rules make the match decision about the behavior described in a report, rather than the apparent importance or internal explanation of the defect.