Benchmark / evidence first
What AI PR review catches in the wild.
Eight real, merged OSS pull requests. Four tools. One consistent review of the diff, its ground truth, and the evidence each tool returned.
Evaluated 2026-06-21 on a curated sample selected 2026-06-20. Read the limits before treating this as a leaderboard.
Scorecard
Per-tool averages, totals, and marked wins
| Tool | Bugs caught (avg) | Signal / noise (avg) | Latency (avg) | False positives (total) | Marked PR wins (total) |
|---|---|---|---|---|---|
| SiftPulse | 50.0% | 0.67 | 19.6s | 2 | 5 / 8 |
| CodeRabbit | 22.5% | 0.40 | 33.6s | 1 | 0 / 8 |
| Greptile | 75.0% | 1.00 | 48.4s | 0 | 2 / 8 |
| Qodo | 16.3% | 0.40 | 23.1s | 0 | 0 / 8 |
Averages are arithmetic per-tool averages across the eight PRs; signal/noise excludes stored nulls. False positives are totals. Wins are counted only from the source score.won flags.
SiftPulse — free to start
See what your team's PRs look like in review
Install the GitHub App in 2 minutes. No credit card. 14-day trial.
Ground truth
Bug: URL parser was stripping port from host when port was default (80/443), causing redirect origin check to pass incorrectly. Exploitable for XSS on sites with non-default ports.
Caught the port-stripping logic error in `lib/adapters/http.js` — the fix replaces `.hostname` with `.host`, which correctly preserves port. Also flagged missing test coverage for non-standard port redirect scenarios.
Full review text
Caught the port-stripping logic error in `lib/adapters/http.js` — the fix replaces `.hostname` with `.host`, which correctly preserves port. Also flagged missing test coverage for non-standard port redirect scenarios.
Flagged the `url.host` vs `url.hostname` issue and noted the security implication (CVE-worthy). Mentioned that the PR description understates the severity. Suggested adding a regression test with a non-standard port.
Full review text
Flagged the `url.host` vs `url.hostname` issue and noted the security implication (CVE-worthy). Mentioned that the PR description understates the severity. Suggested adding a regression test with a non-standard port.
Deep analysis identified the same root cause. Went further — also flagged that the `auth` header can be forwarded to unintended hosts when custom URL is passed. This was NOT in the PR diff; it was a latent bug surfaced by codebase context.
Full review text
Deep analysis identified the same root cause. Went further — also flagged that the `auth` header can be forwarded to unintended hosts when custom URL is passed. This was NOT in the PR diff; it was a latent bug surfaced by codebase context.
Mentioned the `host` vs `hostname` change is significant but did not flag as a security issue. Suggested adding tests. Did not catch the XSS angle.
Full review text
Mentioned the `host` vs `hostname` change is significant but did not flag as a security issue. Suggested adding tests. Did not catch the XSS angle.
Ground truth
Refactor only — no bug. Correctness preserved across all existing test suites. Performance: identical.
Confirmed structural parity. Verified that all call-sites of the extracted utility were updated. Noted one subtle semantic difference: the new module throws a typed `SearchError` instead of generic `Error` — consumers relying on catch blocks will need updating. Flagged 2 affected call-sites.
Full review text
Confirmed structural parity. Verified that all call-sites of the extracted utility were updated. Noted one subtle semantic difference: the new module throws a typed `SearchError` instead of generic `Error` — consumers relying on catch blocks will need updating. Flagged 2 affected call-sites.
Reviewed module-level exports and public API contracts. Found no breaking changes. Suggested adding the new error type to the public exports doc. Clean pass.
Full review text
Reviewed module-level exports and public API contracts. Found no breaking changes. Suggested adding the new error type to the public exports doc. Clean pass.
Used codebase-wide cross-reference to verify no stale imports. Identified 1 call-site in `treeView.ts` that was using a method directly from the old module — not updated. Caught a real regression risk.
Full review text
Used codebase-wide cross-reference to verify no stale imports. Identified 1 call-site in `treeView.ts` that was using a method directly from the old module — not updated. Caught a real regression risk.
Reviewed the diff in isolation. No issues flagged. Did not cross-reference with the rest of the codebase (expected for single-diff analysis).
Full review text
Reviewed the diff in isolation. No issues flagged. Did not cross-reference with the rest of the codebase (expected for single-diff analysis).
Ground truth
Bug: RBAC policy inputs were not sanitized before being passed to the JSON policy engine. Allowed policy injection via specially crafted role names. Patched in this PR.
Caught unsanitized input flowing into `PolicyEngine.Evaluate()`. Flagged the specific line (`policy.go:142`) and explained the injection vector: `${"admin"}}` in role names can cause policy bypass via JSON injection. Recommended `json.Marshal` escaping and a test case with injection payloads.
Full review text
Caught unsanitized input flowing into `PolicyEngine.Evaluate()`. Flagged the specific line (`policy.go:142`) and explained the injection vector: `${"admin"}}` in role names can cause policy bypass via JSON injection. Recommended `json.Marshal` escaping and a test case with injection payloads.
Identified the same injection vector. Noted that the PR description omits the severity — this is a valid CVE submission candidate. Strong pass.
Full review text
Identified the same injection vector. Noted that the PR description omits the severity — this is a valid CVE submission candidate. Strong pass.
Also caught the injection. Additionally cross-referenced the policy engine test suite — found no adversarial test cases existed before this PR. Correctly noted this was a known-gap rather than a new regression.
Full review text
Also caught the injection. Additionally cross-referenced the policy engine test suite — found no adversarial test cases existed before this PR. Correctly noted this was a known-gap rather than a new regression.
Caught the unsanitized input. Provided a good explanation of the attack surface. Suggested escaping the inputs. Solid pass.
Full review text
Caught the unsanitized input. Provided a good explanation of the attack surface. Suggested escaping the inputs. Solid pass.
Ground truth
Performance improvement — reduces initial JS bundle by ~18KB (gzip) on route navigation. Correctness preserved. No regressions in core Web Vitals (LCP, CLS, INP).
Reviewed the chunk-splitting strategy and lazy-loading guard condition. Confirmed chunk boundary is correct. Flagged one potential: if `router.prefetch()` is called in a tight loop (possible in scroll handlers), the new lazy-load could cause staggered re-fetch. Suggested debouncing the guard.
Full review text
Reviewed the chunk-splitting strategy and lazy-loading guard condition. Confirmed chunk boundary is correct. Flagged one potential: if `router.prefetch()` is called in a tight loop (possible in scroll handlers), the new lazy-load could cause staggered re-fetch. Suggested debouncing the guard.
Reviewed perf diff. No correctness concerns. Mentioned LCP impact could be neutral-to-positive. Clean pass.
Full review text
Reviewed perf diff. No correctness concerns. Mentioned LCP impact could be neutral-to-positive. Clean pass.
Cross-referenced all callers of the lazy-loaded chunk and confirmed no synchronous dependency. Performance improvement confirmed by benchmark data in PR. Clean pass.
Full review text
Cross-referenced all callers of the lazy-loaded chunk and confirmed no synchronous dependency. Performance improvement confirmed by benchmark data in PR. Clean pass.
Reviewed diff in isolation. No correctness issues. Suggested a minor test addition for the lazy-load condition. Clean pass.
Full review text
Reviewed diff in isolation. No correctness issues. Suggested a minor test addition for the lazy-load condition. Clean pass.
Ground truth
Architectural migration from in-memory lease tracking to ZooKeeper ephemeral nodes. Fixes a race condition where coordinator failover could cause duplicate segment publish operations. Ground truth confirmed via follow-up bug report filed 3 months later.
Reviewed the ZK session management, ephemeral node creation, and watch re-establishment. Found a gap: if the ZooKeeper connection is briefly interrupted (within sessionTimeout), the watch may not re-fire, causing the lease to appear active when it is not. This exactly matches the follow-up bug report 3 months later. Suggested adding a reconnect watch.
Full review text
Reviewed the ZK session management, ephemeral node creation, and watch re-establishment. Found a gap: if the ZooKeeper connection is briefly interrupted (within sessionTimeout), the watch may not re-fire, causing the lease to appear active when it is not. This exactly matches the follow-up bug report 3 months later. Suggested adding a reconnect watch.
Reviewed the ZK lifecycle. Noted the sessionTimeout handling looks correct. No issues flagged at this time. Clean pass.
Full review text
Reviewed the ZK lifecycle. Noted the sessionTimeout handling looks correct. No issues flagged at this time. Clean pass.
Cross-referenced ZK watch re-firing behavior across the Druid codebase. Identified the same gap: brief connection interruption + no reconnect watch = stale lease. Suggested adding a `getChildren` to re-verify state on reconnect. Excellent catch.
Full review text
Cross-referenced ZK watch re-firing behavior across the Druid codebase. Identified the same gap: brief connection interruption + no reconnect watch = stale lease. Suggested adding a `getChildren` to re-verify state on reconnect. Excellent catch.
Reviewed the diff. Noted the ZooKeeper session management code looks correct. Did not flag the reconnection edge case. Clean pass.
Full review text
Reviewed the diff. Noted the ZooKeeper session management code looks correct. Did not flag the reconnection edge case. Clean pass.
Ground truth
Bug: nested while loops could cause YJIT to allocate the same register to two different live values, producing incorrect bytecode. Fixed by adding an explicit spill before the inner loop.
Reviewed the register spill logic. Confirmed the fix addresses the live-range collision. Noted that the same pattern could appear in `until` loops as well — flagged a pre-emptive suggestion to audit `until` for the same issue.
Full review text
Reviewed the register spill logic. Confirmed the fix addresses the live-range collision. Noted that the same pattern could appear in `until` loops as well — flagged a pre-emptive suggestion to audit `until` for the same issue.
Reviewed the fix. Noted that register allocation for nested loops is complex and the change looks correct. Did not identify the `until` risk.
Full review text
Reviewed the fix. Noted that register allocation for nested loops is complex and the change looks correct. Did not identify the `until` risk.
Used cross-reference across all YJIT loop handling code. Identified the `until` case as potentially affected (same register spill issue). Correctly flagged this as a pre-emptive finding.
Full review text
Used cross-reference across all YJIT loop handling code. Identified the `until` case as potentially affected (same register spill issue). Correctly flagged this as a pre-emptive finding.
Reviewed the tiny diff. Found no correctness issues. Pass.
Full review text
Reviewed the tiny diff. Found no correctness issues. Pass.
Ground truth
Dependency security patch — no code logic change. The `send` package CVE was real; this is the correct fix.
Verified `send` 0.19.0 changelog against the CVE. Confirmed the patch addresses the reported vulnerability (path traversal in `send.static`). Noted that `express.static()` middleware still depends on `send` — the fix covers this case. No further action needed.
Full review text
Verified `send` 0.19.0 changelog against the CVE. Confirmed the patch addresses the reported vulnerability (path traversal in `send.static`). Noted that `express.static()` middleware still depends on `send` — the fix covers this case. No further action needed.
Reviewed the dependency upgrade. Confirmed it is a direct CVE patch. Clean pass.
Full review text
Reviewed the dependency upgrade. Confirmed it is a direct CVE patch. Clean pass.
Cross-referenced the CVE report against the send 0.19.0 changelog. Verified the path traversal fix is included. Confirmed no breaking changes in 0.19.0 that affect Express. Clean pass.
Full review text
Cross-referenced the CVE report against the send 0.19.0 changelog. Verified the path traversal fix is included. Confirmed no breaking changes in 0.19.0 that affect Express. Clean pass.
Reviewed the dep bump. No issues. Pass.
Full review text
Reviewed the dep bump. No issues. Pass.
Ground truth
TypeScript strict mode migration — no runtime behavior change. Correctness preserved. Some TypeScript errors remain after this PR (known, tracked separately).
Reviewed type changes across all loader signatures. Confirmed `RequestLoaderArgs` and `LoaderFunctionArgs` are correctly applied. Found 2 loaders that cast `any` to bypass strict mode instead of properly typing — flagged these as technical debt. No runtime correctness issues.
Full review text
Reviewed type changes across all loader signatures. Confirmed `RequestLoaderArgs` and `LoaderFunctionArgs` are correctly applied. Found 2 loaders that cast `any` to bypass strict mode instead of properly typing — flagged these as technical debt. No runtime correctness issues.
Reviewed type changes. Noted the `as any` casts but did not flag them as issues. Clean pass.
Full review text
Reviewed type changes. Noted the `as any` casts but did not flag them as issues. Clean pass.
Cross-referenced all loader call-sites in the codebase. Found 1 call-site that passes a plain object to a loader expecting `LoaderArgs<typeof loader>` — a type mismatch that TypeScript did not catch because the receiving loader uses `any`. This is a latent bug.
Full review text
Cross-referenced all loader call-sites in the codebase. Found 1 call-site that passes a plain object to a loader expecting `LoaderArgs<typeof loader>` — a type mismatch that TypeScript did not catch because the receiving loader uses `any`. This is a latent bug.
Reviewed the TypeScript changes in isolation. No issues flagged. Did not catch the `as any` casts or the cross-referenced type mismatch.
Full review text
Reviewed the TypeScript changes in isolation. No issues flagged. Did not catch the `as any` casts or the cross-referenced type mismatch.
Method / limitations / source
How to read this benchmark
The cards below preserve the review text and scores for every PR × tool combination. Each PR title links to the public source pull request; expand the ground truth and full review text to audit the judgment behind the numbers.
01 Dataset
- Public, merged PRs from well-known OSS repos
- Must have a follow-up commit or linked issue establishing ground-truth
- Mix of diff sizes and categories: bugfix, refactor, security, performance, large, tiny, dep bump, migration
- Excluded: WIP PRs, PRs with no review activity, ambiguous ground truth
Selected on 2026-06-20. All 8 PRs are from distinct repos to avoid clustering artifacts; the source PR link and ground truth appear on each evidence card.
02 Test conditions
| Tool | Version / Account | Run date | Method |
|---|---|---|---|
| CodeRabbit | CR-2024.6 pro |
2026-06-21 | GitHub App install on test repo |
| Greptile | production (app.greptile.ai) pro |
2026-06-21 | API with full codebase indexing |
| Qodo | Qodo Merge v3.1 14-day trial |
2026-06-21 | GitHub App install on test repo |
| SiftPulse | self-hosted OpenAI GPT-4o |
2026-06-21 | GitHub App install on test repo |
Note: Competitor accounts were provisioned fresh on 2026-06-20. Version numbers reflect what was available at time of run. Newer versions may perform differently.
03 Scoring
- bugs_caught
- Percentage of ground-truth issues surfaced in the tool review comment (vs. ground truth from follow-up commit/issue). 0–100%.
- false_positives
- Count of findings that, upon human review of the diff, turned out to be noise.
- signal_noise
- bugs_caught / (bugs_caught + false_positives). Measures how clean the signal is. 1.0 = perfect.
- latency_seconds
- Wall-clock from diff submission to first review comment posted. Lower is better.
- wins
- Count of PRs marked won in the source score data. A PR with no marked winner is not assigned to a tool.
Raw outputs JSON: /api/benchmark/data — includes full review text for every PR × tool combination.
04 Limitations
- This is a curated sample of eight PRs, not a representative benchmark of all repositories or change types.
- The evaluation was run by SiftPulse engineering; it was not an independent or blind third-party study.
- Tool versions, account tiers, and run dates are dated conditions; newer releases or different accounts may perform differently.
- Greptile had full-codebase indexing, so context was not identical across tools on cross-file cases.
- Finding labels, false positives, and ground truth were hand-reviewed judgments based on the linked PR evidence.
Last updated 2026-06-21.
05 Honest results
SiftPulse has 5 marked wins in this sample. It did not receive the marked win on the two cases where Greptile did:
-
PR #2: microsoft/vscode refactor: Greptile caught a stale call-site in
treeView.tsthrough full-codebase indexing. That is a structural advantage for large refactors; SiftPulse reviews the PR-level context represented here. - PR #8: remix-run/remix TypeScript migration: Greptile cross-referenced loader call-sites and caught a type mismatch. The source data marks this as Greptile's win; this page does not claim a SiftPulse win where none is stored.
These results describe this eight-PR sample and the documented context difference. If full-codebase, cross-file analysis is your priority, inspect the raw reviews and choose accordingly.
Run the benchmark on your team's PRs
Install SiftPulse on any public or private repo. 14-day free trial, no credit card.