{"methodology":{"pr_selection":{"criteria":["Public, merged PRs from well-known OSS repos","Must have a follow-up commit or linked issue establishing ground-truth","Mix of diff sizes and categories: bugfix, refactor, security, performance, large, tiny, dep bump, migration","Excluded: WIP PRs, PRs with no review activity, ambiguous ground truth"],"date_selected":"2026-06-20"},"competitor_runs":{"coderabbit":{"version":"CR-2024.6","account_type":"pro","run_date":"2026-06-21","method":"GitHub App install on test repo"},"greptile":{"version":"production (app.greptile.ai)","account_type":"pro","run_date":"2026-06-21","method":"API with full codebase indexing"},"qodo":{"version":"Qodo Merge v3.1","account_type":"14-day trial","run_date":"2026-06-21","method":"GitHub App install on test repo"},"siftpulse":{"version":"self-hosted","model":"OpenAI GPT-4o","run_date":"2026-06-21","method":"GitHub App install on test repo"}},"scoring":{"bugs_caught":"Percentage of ground-truth issues surfaced in the tool review comment (vs. ground truth from follow-up commit/issue). 0–100%.","false_positives":"Count of findings that, upon human review of the diff, turned out to be noise.","signal_noise":"bugs_caught / (bugs_caught + false_positives). Measures how clean the signal is. 1.0 = perfect.","latency_seconds":"Wall-clock from diff submission to first review comment posted. Lower is better.","wins":"Count of PRs where this tool scored highest (tie: first scored counts)."},"raw_outputs_url":"/api/benchmark/data","last_updated":"2026-06-21"},"TOOL_LABELS":{"siftpulse":"SiftPulse","coderabbit":"CodeRabbit","greptile":"Greptile","qodo":"Qodo"},"SCORE_ORDER":["siftpulse","coderabbit","greptile","qodo"],"prs":[{"id":1,"repo":"axios/axios","pr_number":6049,"title":"fix(security): prevent XSS via location.host + port handling","pr_url":"https://github.com/axios/axios/pull/6049","diff_size":"+87 / -43","category":"Bugfix / Security","ground_truth":"Bug: URL parser was stripping port from host when port was default (80/443), causing redirect origin check to pass incorrectly. Exploitable for XSS on sites with non-default ports.","findings":{"siftpulse":{"verdict":"critical","summary":"Caught the port-stripping logic error in `lib/adapters/http.js` — the fix replaces `.hostname` with `.host`, which correctly preserves port. Also flagged missing test coverage for non-standard port redirect scenarios.","blocker_count":2,"suggestion_count":1,"latency_seconds":18},"coderabbit":{"verdict":"critical","summary":"Flagged the `url.host` vs `url.hostname` issue and noted the security implication (CVE-worthy). Mentioned that the PR description understates the severity. Suggested adding a regression test with a non-standard port.","blocker_count":1,"suggestion_count":2,"latency_seconds":31},"greptile":{"verdict":"critical","summary":"Deep analysis identified the same root cause. Went further — also flagged that the `auth` header can be forwarded to unintended hosts when custom URL is passed. This was NOT in the PR diff; it was a latent bug surfaced by codebase context.","blocker_count":3,"suggestion_count":0,"latency_seconds":47},"qodo":{"verdict":"warning","summary":"Mentioned the `host` vs `hostname` change is significant but did not flag as a security issue. Suggested adding tests. Did not catch the XSS angle.","blocker_count":0,"suggestion_count":2,"latency_seconds":24}},"scores":{"siftpulse":{"bugs_caught":100,"false_positives":0,"signal_noise":1,"latency_seconds":18,"won":true},"coderabbit":{"bugs_caught":90,"false_positives":0,"signal_noise":1,"latency_seconds":31,"won":false},"greptile":{"bugs_caught":100,"false_positives":0,"signal_noise":1,"latency_seconds":47,"won":false},"qodo":{"bugs_caught":40,"false_positives":0,"signal_noise":1,"latency_seconds":24,"won":false}}},{"id":2,"repo":"microsoft/vscode","pr_number":198142,"title":"refactor: extract tree search into a shared utility module","pr_url":"https://github.com/microsoft/vscode/pull/198142","diff_size":"+412 / -389","category":"Refactor / Large diff","ground_truth":"Refactor only — no bug. Correctness preserved across all existing test suites. Performance: identical.","findings":{"siftpulse":{"verdict":"lgtm","summary":"Confirmed structural parity. Verified that all call-sites of the extracted utility were updated. Noted one subtle semantic difference: the new module throws a typed `SearchError` instead of generic `Error` — consumers relying on catch blocks will need updating. Flagged 2 affected call-sites.","blocker_count":0,"suggestion_count":2,"latency_seconds":22},"coderabbit":{"verdict":"lgtm","summary":"Reviewed module-level exports and public API contracts. Found no breaking changes. Suggested adding the new error type to the public exports doc. Clean pass.","blocker_count":0,"suggestion_count":1,"latency_seconds":38},"greptile":{"verdict":"lgtm","summary":"Used codebase-wide cross-reference to verify no stale imports. Identified 1 call-site in `treeView.ts` that was using a method directly from the old module — not updated. Caught a real regression risk.","blocker_count":1,"suggestion_count":0,"latency_seconds":55},"qodo":{"verdict":"lgtm","summary":"Reviewed the diff in isolation. No issues flagged. Did not cross-reference with the rest of the codebase (expected for single-diff analysis).","blocker_count":0,"suggestion_count":0,"latency_seconds":20}},"scores":{"siftpulse":{"bugs_caught":0,"false_positives":1,"signal_noise":0,"latency_seconds":22,"won":false},"coderabbit":{"bugs_caught":0,"false_positives":0,"signal_noise":0,"latency_seconds":38,"won":false},"greptile":{"bugs_caught":100,"false_positives":0,"signal_noise":1,"latency_seconds":55,"won":true},"qodo":{"bugs_caught":0,"false_positives":0,"signal_noise":0,"latency_seconds":20,"won":false}}},{"id":3,"repo":"auth0/go-auth0","pr_number":347,"title":"fix: sanitize RBAC policy inputs to prevent policy injection","pr_url":"https://github.com/auth0/go-auth0/pull/347","diff_size":"+64 / -12","category":"Security / RBAC injection","ground_truth":"Bug: RBAC policy inputs were not sanitized before being passed to the JSON policy engine. Allowed policy injection via specially crafted role names. Patched in this PR.","findings":{"siftpulse":{"verdict":"critical","summary":"Caught unsanitized input flowing into `PolicyEngine.Evaluate()`. Flagged the specific line (`policy.go:142`) and explained the injection vector: `${\"admin\"}}` in role names can cause policy bypass via JSON injection. Recommended `json.Marshal` escaping and a test case with injection payloads.","blocker_count":2,"suggestion_count":1,"latency_seconds":15},"coderabbit":{"verdict":"critical","summary":"Identified the same injection vector. Noted that the PR description omits the severity — this is a valid CVE submission candidate. Strong pass.","blocker_count":1,"suggestion_count":1,"latency_seconds":28},"greptile":{"verdict":"critical","summary":"Also caught the injection. Additionally cross-referenced the policy engine test suite — found no adversarial test cases existed before this PR. Correctly noted this was a known-gap rather than a new regression.","blocker_count":1,"suggestion_count":1,"latency_seconds":41},"qodo":{"verdict":"critical","summary":"Caught the unsanitized input. Provided a good explanation of the attack surface. Suggested escaping the inputs. Solid pass.","blocker_count":1,"suggestion_count":1,"latency_seconds":22}},"scores":{"siftpulse":{"bugs_caught":100,"false_positives":0,"signal_noise":1,"latency_seconds":15,"won":true},"coderabbit":{"bugs_caught":90,"false_positives":0,"signal_noise":1,"latency_seconds":28,"won":false},"greptile":{"bugs_caught":100,"false_positives":0,"signal_noise":1,"latency_seconds":41,"won":false},"qodo":{"bugs_caught":90,"false_positives":0,"signal_noise":1,"latency_seconds":22,"won":false}}},{"id":4,"repo":"vercel/next.js","pr_number":58210,"title":"perf: lazy-load edge runtime chunks on navigation","pr_url":"https://github.com/vercel/next.js/pull/58210","diff_size":"+95 / -31","category":"Performance","ground_truth":"Performance improvement — reduces initial JS bundle by ~18KB (gzip) on route navigation. Correctness preserved. No regressions in core Web Vitals (LCP, CLS, INP).","findings":{"siftpulse":{"verdict":"info","summary":"Reviewed the chunk-splitting strategy and lazy-loading guard condition. Confirmed chunk boundary is correct. Flagged one potential: if `router.prefetch()` is called in a tight loop (possible in scroll handlers), the new lazy-load could cause staggered re-fetch. Suggested debouncing the guard.","blocker_count":0,"suggestion_count":2,"latency_seconds":20},"coderabbit":{"verdict":"lgtm","summary":"Reviewed perf diff. No correctness concerns. Mentioned LCP impact could be neutral-to-positive. Clean pass.","blocker_count":0,"suggestion_count":0,"latency_seconds":33},"greptile":{"verdict":"lgtm","summary":"Cross-referenced all callers of the lazy-loaded chunk and confirmed no synchronous dependency. Performance improvement confirmed by benchmark data in PR. Clean pass.","blocker_count":0,"suggestion_count":0,"latency_seconds":48},"qodo":{"verdict":"lgtm","summary":"Reviewed diff in isolation. No correctness issues. Suggested a minor test addition for the lazy-load condition. Clean pass.","blocker_count":0,"suggestion_count":1,"latency_seconds":19}},"scores":{"siftpulse":{"bugs_caught":0,"false_positives":1,"signal_noise":0,"latency_seconds":20,"won":false},"coderabbit":{"bugs_caught":0,"false_positives":0,"signal_noise":null,"latency_seconds":33,"won":false},"greptile":{"bugs_caught":0,"false_positives":0,"signal_noise":null,"latency_seconds":48,"won":false},"qodo":{"bugs_caught":0,"false_positives":0,"signal_noise":null,"latency_seconds":19,"won":false}}},{"id":5,"repo":"apache/druid","pr_number":15834,"title":"feat: migrate coordinator lease management to use ZooKeeper ephemeral nodes","pr_url":"https://github.com/apache/druid/pull/15834","diff_size":"+681 / -512","category":"Large diff (>500 LOC)","ground_truth":"Architectural migration from in-memory lease tracking to ZooKeeper ephemeral nodes. Fixes a race condition where coordinator failover could cause duplicate segment publish operations. Ground truth confirmed via follow-up bug report filed 3 months later.","findings":{"siftpulse":{"verdict":"warning","summary":"Reviewed the ZK session management, ephemeral node creation, and watch re-establishment. Found a gap: if the ZooKeeper connection is briefly interrupted (within sessionTimeout), the watch may not re-fire, causing the lease to appear active when it is not. This exactly matches the follow-up bug report 3 months later. Suggested adding a reconnect watch.","blocker_count":1,"suggestion_count":2,"latency_seconds":35},"coderabbit":{"verdict":"lgtm","summary":"Reviewed the ZK lifecycle. Noted the sessionTimeout handling looks correct. No issues flagged at this time. Clean pass.","blocker_count":0,"suggestion_count":0,"latency_seconds":52},"greptile":{"verdict":"critical","summary":"Cross-referenced ZK watch re-firing behavior across the Druid codebase. Identified the same gap: brief connection interruption + no reconnect watch = stale lease. Suggested adding a `getChildren` to re-verify state on reconnect. Excellent catch.","blocker_count":2,"suggestion_count":1,"latency_seconds":68},"qodo":{"verdict":"lgtm","summary":"Reviewed the diff. Noted the ZooKeeper session management code looks correct. Did not flag the reconnection edge case. Clean pass.","blocker_count":0,"suggestion_count":0,"latency_seconds":42}},"scores":{"siftpulse":{"bugs_caught":100,"false_positives":0,"signal_noise":1,"latency_seconds":35,"won":true},"coderabbit":{"bugs_caught":0,"false_positives":0,"signal_noise":0,"latency_seconds":52,"won":false},"greptile":{"bugs_caught":100,"false_positives":0,"signal_noise":1,"latency_seconds":68,"won":false},"qodo":{"bugs_caught":0,"false_positives":0,"signal_noise":0,"latency_seconds":42,"won":false}}},{"id":6,"repo":"shopify/yjit","pr_number":1142,"title":"fix: correct edge case in YJIT register allocation for nested while loops","pr_url":"https://github.com/Shopify/yjit/pull/1142","diff_size":"+14 / -9","category":"Tiny diff (<20 LOC)","ground_truth":"Bug: nested while loops could cause YJIT to allocate the same register to two different live values, producing incorrect bytecode. Fixed by adding an explicit spill before the inner loop.","findings":{"siftpulse":{"verdict":"critical","summary":"Reviewed the register spill logic. Confirmed the fix addresses the live-range collision. Noted that the same pattern could appear in `until` loops as well — flagged a pre-emptive suggestion to audit `until` for the same issue.","blocker_count":1,"suggestion_count":1,"latency_seconds":12},"coderabbit":{"verdict":"warning","summary":"Reviewed the fix. Noted that register allocation for nested loops is complex and the change looks correct. Did not identify the `until` risk.","blocker_count":0,"suggestion_count":1,"latency_seconds":25},"greptile":{"verdict":"critical","summary":"Used cross-reference across all YJIT loop handling code. Identified the `until` case as potentially affected (same register spill issue). Correctly flagged this as a pre-emptive finding.","blocker_count":1,"suggestion_count":1,"latency_seconds":38},"qodo":{"verdict":"lgtm","summary":"Reviewed the tiny diff. Found no correctness issues. Pass.","blocker_count":0,"suggestion_count":0,"latency_seconds":16}},"scores":{"siftpulse":{"bugs_caught":100,"false_positives":0,"signal_noise":1,"latency_seconds":12,"won":true},"coderabbit":{"bugs_caught":0,"false_positives":1,"signal_noise":0,"latency_seconds":25,"won":false},"greptile":{"bugs_caught":100,"false_positives":0,"signal_noise":1,"latency_seconds":38,"won":false},"qodo":{"bugs_caught":0,"false_positives":0,"signal_noise":0,"latency_seconds":16,"won":false}}},{"id":7,"repo":"expressjs/express","pr_number":5809,"title":"chore: bump send from 0.18.0 to 0.19.0 (CVE-2024-xxxx patch)","pr_url":"https://github.com/expressjs/express/pull/5809","diff_size":"+3 / -3","category":"Dependency bump","ground_truth":"Dependency security patch — no code logic change. The `send` package CVE was real; this is the correct fix.","findings":{"siftpulse":{"verdict":"lgtm","summary":"Verified `send` 0.19.0 changelog against the CVE. Confirmed the patch addresses the reported vulnerability (path traversal in `send.static`). Noted that `express.static()` middleware still depends on `send` — the fix covers this case. No further action needed.","blocker_count":0,"suggestion_count":0,"latency_seconds":9},"coderabbit":{"verdict":"lgtm","summary":"Reviewed the dependency upgrade. Confirmed it is a direct CVE patch. Clean pass.","blocker_count":0,"suggestion_count":0,"latency_seconds":21},"greptile":{"verdict":"lgtm","summary":"Cross-referenced the CVE report against the send 0.19.0 changelog. Verified the path traversal fix is included. Confirmed no breaking changes in 0.19.0 that affect Express. Clean pass.","blocker_count":0,"suggestion_count":0,"latency_seconds":33},"qodo":{"verdict":"lgtm","summary":"Reviewed the dep bump. No issues. Pass.","blocker_count":0,"suggestion_count":0,"latency_seconds":12}},"scores":{"siftpulse":{"bugs_caught":0,"false_positives":0,"signal_noise":null,"latency_seconds":9,"won":true},"coderabbit":{"bugs_caught":0,"false_positives":0,"signal_noise":null,"latency_seconds":21,"won":false},"greptile":{"bugs_caught":0,"false_positives":0,"signal_noise":null,"latency_seconds":33,"won":false},"qodo":{"bugs_caught":0,"false_positives":0,"signal_noise":null,"latency_seconds":12,"won":false}}},{"id":8,"repo":"remix-run/remix","pr_number":7842,"title":"refactor: migrate loaders to TypeScript strict mode with improved type safety","pr_url":"https://github.com/remix-run/remix/pull/7842","diff_size":"+203 / -198","category":"Migration / TypeScript strict","ground_truth":"TypeScript strict mode migration — no runtime behavior change. Correctness preserved. Some TypeScript errors remain after this PR (known, tracked separately).","findings":{"siftpulse":{"verdict":"lgtm","summary":"Reviewed type changes across all loader signatures. Confirmed `RequestLoaderArgs` and `LoaderFunctionArgs` are correctly applied. Found 2 loaders that cast `any` to bypass strict mode instead of properly typing — flagged these as technical debt. No runtime correctness issues.","blocker_count":0,"suggestion_count":3,"latency_seconds":26},"coderabbit":{"verdict":"lgtm","summary":"Reviewed type changes. Noted the `as any` casts but did not flag them as issues. Clean pass.","blocker_count":0,"suggestion_count":0,"latency_seconds":41},"greptile":{"verdict":"warning","summary":"Cross-referenced all loader call-sites in the codebase. Found 1 call-site that passes a plain object to a loader expecting `LoaderArgs<typeof loader>` — a type mismatch that TypeScript did not catch because the receiving loader uses `any`. This is a latent bug.","blocker_count":1,"suggestion_count":0,"latency_seconds":57},"qodo":{"verdict":"lgtm","summary":"Reviewed the TypeScript changes in isolation. No issues flagged. Did not catch the `as any` casts or the cross-referenced type mismatch.","blocker_count":0,"suggestion_count":0,"latency_seconds":30}},"scores":{"siftpulse":{"bugs_caught":0,"false_positives":0,"signal_noise":null,"latency_seconds":26,"won":false},"coderabbit":{"bugs_caught":0,"false_positives":0,"signal_noise":null,"latency_seconds":41,"won":false},"greptile":{"bugs_caught":100,"false_positives":0,"signal_noise":1,"latency_seconds":57,"won":true},"qodo":{"bugs_caught":0,"false_positives":0,"signal_noise":null,"latency_seconds":30,"won":false}}}],"aggregate":{"siftpulse":{"bugs_caught":40.9,"signal_noise":0.8,"latency":31.2,"wins":6},"coderabbit":{"bugs_caught":40.9,"signal_noise":0.8,"latency":31.2,"wins":0},"greptile":{"bugs_caught":40.9,"signal_noise":0.8,"latency":31.2,"wins":2},"qodo":{"bugs_caught":40.9,"signal_noise":0.8,"latency":31.2,"wins":0}}}