AI-assisted code review is becoming an increasingly common part of the software-development workflow.
The basic promise is compelling: give an AI system a pull request, let it inspect the relevant code and context, and have it identify bugs, security issues, regressions, or other problems before a human reviewer has to.
But evaluating these systems is harder than demonstrating them.
A vendor can report that its model catches a certain percentage of bugs on a benchmark. Another vendor can publish a higher number. A third can show an impressive collection of screenshots from production pull requests.
Those numbers may all be legitimate while still failing to answer the question an engineering organization actually cares about:
How well does this AI reviewer work on our code, with our pull requests, under our review process?
This article proposes a reproducible methodology for answering that question.
The objective isn't to produce a universal leaderboard. It's to give engineering teams a framework they can run independently, compare over time, and improve publicly.
Start with the evaluation question
Before collecting data, define what you're trying to measure.
There are several different questions hiding inside "How good is this AI code reviewer?"
For example:
Does it detect real bugs?
Does it avoid reporting things that aren't bugs?
Does it understand severity?
Does it produce useful, actionable explanations?
Do developers trust its findings?
How quickly does it review a PR?
Does it actually reduce human review effort?
These aren't interchangeable.
A system can have excellent bug detection but terrible precision. It can have excellent precision while missing most serious vulnerabilities. It can identify problems accurately but take too long to be useful in CI.
That's why I would treat AI code review as a multi-dimensional evaluation problem, rather than a single accuracy metric.
1. Construct a benchmark dataset
The dataset is the foundation of the entire evaluation.
If the dataset is unrealistic, every downstream metric becomes questionable.
I'd build a dataset from several sources:
Historical production bugs
Security fixes
Reverted changes
Incident-related pull requests
Normal feature development
Refactoring
Dependency updates
Performance changes
API changes
Database/schema migrations
Test changes
Clean PRs with no known defects
The dataset should contain both positive cases and negative cases.
Positive cases contain known issues.
Negative cases are PRs where there is no known issue that the AI reviewer is expected to report.
The negative cases are particularly important because otherwise you can measure recall without measuring whether the system produces excessive noise.
Preserve the original review context
Ideally, a benchmark PR should resemble the state in which the original review occurred.
Capture:
PR description
Diff
Base revision
Changed files
Repository state
Tests
Configuration
Relevant documentation
Existing review comments, if they're part of the intended context
Then explicitly document which of these artifacts the AI reviewer is allowed to access.
This is important for reproducibility.
A system that can inspect an entire repository is being evaluated differently from one that only receives the diff.
Neither approach is inherently wrong. The benchmark simply needs to make the distinction explicit.
2. Prefer real-world bugs, but don't eliminate synthetic cases
Real-world bugs should be the backbone of the benchmark.
They represent the problems engineers actually encounter, including messy interactions between components that are difficult to anticipate when constructing artificial examples.
Useful sources include:
Postmortems
Security advisories
Bug trackers
Regression fixes
Production incidents
Historical pull requests
Reverted commits
However, synthetic bugs still have value.
Synthetic cases let you deliberately test categories that may be underrepresented in your historical data.
For example, you might want to test:
SQL injection
Authorization bypass
Race conditions
Integer overflow
Incorrect caching
Resource leaks
Path traversal
Null handling
Concurrency errors
The key is to keep the two categories separate.
Report something like:
| Dataset | PRs | Known bugs |
|---|---|---|
| Historical | 250 | 117 |
| Synthetic | 100 | 100 |
| Clean | 150 | 0 |
Don't silently combine them into one number.
Synthetic benchmarks can measure targeted capabilities. Historical benchmarks provide stronger evidence about real-world performance.
3. Select PRs deliberately
A random sample is better than cherry-picking impressive examples, but pure random sampling may not be enough.
Rare but important bugs can disappear in a random dataset.
I'd use stratified sampling.
For example, stratify by:
Repository
Language
PR size
Change type
Severity
Bug category
Team/domain
A dataset might look like:
| Category | Target |
|---|---|
| Small PRs | 35% |
| Medium PRs | 45% |
| Large PRs | 20% |
| Security-related | 15% |
| Reliability | 20% |
| Correctness | 35% |
| Performance | 10% |
| Other | 20% |
These percentages are examples, not universal recommendations.
The point is to make the sampling strategy explicit.
Also record the selection criteria before running the evaluation.
Otherwise, it becomes very easy to unconsciously construct a benchmark around cases you expect the tool to perform well on.
4. Test repository and language diversity
A reviewer shouldn't be evaluated only against the repository where it happens to look best.
If your organization operates:
payments-api Go
web-app TypeScript
data-platform Python
mobile-service Kotlin
infrastructure Terraform
then the benchmark should reflect that environment.
Language diversity matters because different languages expose different classes of problems and provide different amounts of semantic information to analysis systems.
Repository architecture matters too.
Compare:
Monorepos
Small services
Large services
Libraries
Frontend applications
Infrastructure repositories
Data pipelines
A tool may behave very differently depending on whether it is analyzing a 500-line service or a massive monorepo with hundreds of dependencies.
Don't necessarily demand equal representation. Instead, report the distribution.
5. Control reviewer context
This is one of the easiest ways to accidentally invalidate a comparison.
Every system under evaluation should receive equivalent information.
If Reviewer A gets:
PR diff
+
entire repository
+
documentation
+
test results
while Reviewer B receives only:
PR diff
their scores aren't directly comparable.
Define a benchmark context contract.
For example:
Repository: full checkout
PR: complete diff
Tests: available
CI results: available
Documentation: repository documentation available
External internet: disabled
Previous review comments: unavailable
Then use that same contract wherever technically possible.
If a product fundamentally requires a different context model, document the difference rather than pretending the evaluations are identical.
6. Build explicit ground truth
Ground truth is harder than it sounds.
A historical bug doesn't automatically tell you exactly what an AI reviewer should have reported.
For every known issue, create a structured annotation.
For example:
Issue ID: AUTH-042
Repository: payments-api
PR: #1842
Bug class: Authorization
Severity: Critical
Affected code:
internal/export/service.go:81-94
Expected finding:
Export operation checks authentication but not
the user's export permission.
Required evidence:
Reviewer must identify the missing authorization
condition and explain the resulting access control failure.
Ideally, have at least two experienced engineers independently label important cases.
Then resolve disagreements.
This gives you something more defensible than a single person's interpretation of what the AI "should have" noticed.
7. Measure bug detection rate
The obvious metric is:
Detection rate = correctly identified known bugs / known bugs
If there are 100 known bugs and the reviewer correctly identifies 72, detection rate is 72%.
Useful as that is, it should be reported with more detail.
Break it down by:
Severity
Language
Repository
Bug class
PR size
Real vs synthetic
For example:
| Category | Bugs | Found | Detection |
|---|---|---|---|
| Critical | 20 | 18 | 90% |
| High | 35 | 25 | 71% |
| Medium | 45 | 29 | 64% |
| Total | 100 | 72 | 72% |
Now the number tells us something useful.
A 72% overall detection rate hides a very different story if the system finds 90% of critical issues but only 40% of low-severity issues—or vice versa.
8. Measure severity accuracy
Detection alone isn't enough.
Suppose the benchmark contains a critical authorization bypass and the reviewer reports:
"Potential maintainability issue."
Did it detect the problem?
Maybe.
Did it correctly assess its importance?
No.
Record both the ground-truth severity and the reviewer's predicted severity.
A simple metric is the percentage of findings whose severity falls into the correct category.
You can also use a confusion matrix:
| Predicted / Actual | Critical | High | Medium | Low |
|---|---|---|---|---|
| Critical | 18 | 3 | 0 | 0 |
| High | 2 | 22 | 5 | 1 |
| Medium | 0 | 7 | 31 | 8 |
| Low | 0 | 1 | 9 | 13 |
This reveals systematic behavior.
For example, a reviewer might consistently underestimate security issues.
That is a much more useful finding than simply saying "severity accuracy = 76%."
9. Measure false positives
Now evaluate the clean PRs.
If the AI reports five problems on every clean PR, developers will quickly learn to ignore it.
A useful metric is precision:
Precision = correct findings / all findings
Suppose the reviewer produces:
Correct findings: 80
Incorrect findings: 40
Precision is:
80 / 120 = 66.7%
Also report the simpler operational metric:
False findings per PR
This is often easier for engineering leaders and developers to understand.
For example:
Reviewer A: 0.8 false findings / PR
Reviewer B: 2.9 false findings / PR
Even if both have similar bug detection rates, their impact on developers could be dramatically different.
10. Deduplicate findings
AI reviewers can produce several comments about one underlying defect.
Consider a missing authorization check.
The system might report:
Missing permission validation.
User could access unauthorized records.
Consider checking account ownership.
Export endpoint should validate privileges.
These may all represent the same issue.
If you count them as four successful findings, you're inflating the score.
Define a finding identity.
Multiple comments should count as one finding when they refer to the same underlying defect and would result in the same engineering action.
Then measure:
Duplicate rate = duplicate findings / total findings
Also report noisy but technically true findings separately from actual false positives.
That distinction matters when evaluating developer experience.
11. Measure developer acceptance
Automated evaluation tells you whether a finding corresponds to ground truth.
It doesn't tell you whether developers find the comment useful.
Run a blinded human evaluation on a sample of findings.
Ask developers:
Is this a real issue?
Is the explanation correct?
Is it actionable?
Would you change the code?
Would you keep this comment in a production review?
You can define:
Acceptance rate = accepted findings / findings evaluated
For a live pilot, measure actual outcomes as well:
Accepted
Fixed
Dismissed
Ignored
Marked duplicate
This bridges the gap between benchmark correctness and real-world utility.
A technically correct finding that nobody can act on may have less practical value than a concise finding that consistently leads to fixes.
12. Measure time-to-review
Latency is another dimension that tends to disappear from accuracy benchmarks.
Record:
Time from PR submission to review start
Model processing time
Time to first finding
Time to complete review
P50 latency
P95 latency
P99 latency
Measure latency by PR size too.
For example:
| PR size | P50 | P95 |
|---|---|---|
| Small | 45 sec | 2 min |
| Medium | 2 min | 6 min |
| Large | 7 min | 18 min |
A reviewer that produces excellent findings but routinely blocks the development workflow for 20 minutes may not be suitable for synchronous CI review.
13. Estimate developer time saved
This is perhaps the hardest metric—and one of the most important.
The goal isn't to claim that every AI finding replaces human review.
Instead, estimate the incremental work the system removes.
One approach is to ask reviewers after sampled PRs:
"How much time would you otherwise have spent identifying or validating this issue?"
Then compare that with the time required to inspect AI findings.
For example:
AI review time: 3.5 hours
Developer triage time: 5.0 hours
Estimated avoided review work: 11.0 hours
Net estimated time saved: 6.0 hours
These are estimates, so report methodology and uncertainty.
For a stronger experiment, compare two groups:
Control:
Human review without AI
Treatment:
Human review with AI
Measure review duration, defects found, review comments, and subsequent corrections.
That gives you better evidence than asking developers whether they "feel faster."
14. A proposed scoring model
A simple weighted score can make results easier to compare.
For example:
| Metric | Weight |
|---|---|
| Bug detection | 30% |
| False-positive/precision score | 20% |
| Severity accuracy | 10% |
| Developer acceptance | 15% |
| Duplicate/noise score | 10% |
| Review latency | 5% |
| Developer time saved | 10% |
| Total | 100% |
Normalize each metric to 0–100 and calculate:
Overall score = Σ(metric score × weight)
An example evaluation could look like this:
| Metric | Result | Normalized | Weight | Weighted |
|---|---|---|---|---|
| Bug detection | 78% | 78 | 30% | 23.4 |
| Precision | 72% | 72 | 20% | 14.4 |
| Severity accuracy | 81% | 81 | 10% | 8.1 |
| Acceptance | 69% | 69 | 15% | 10.4 |
| Duplicate/noise | 88% | 88 | 10% | 8.8 |
| Latency | 92 | 92 | 5% | 4.6 |
| Time saved | 63 | 63 | 10% | 6.3 |
| Overall | 100% | 76.0 |
The number is useful as a summary, not as the conclusion.
Why the scoring model has limitations
There is no objectively correct weighting.
Why should bug detection be worth 30% rather than 40%?
Why should latency be worth 5%?
The answer depends on the organization.
A security-sensitive company might give critical-vulnerability recall substantially more weight.
A large organization processing thousands of PRs per day might care much more about latency and false-positive rates.
There are also problems with collapsing everything into one number.
Imagine two systems:
System A
Excellent critical-bug recall
Moderate precision
Fast
System B
Excellent precision
Poor critical-bug recall
Very fast
An overall score might make them look nearly identical even though their risk profiles are completely different.
For that reason, I would use hard gates alongside the weighted score.
For example:
Overall score >= 75
AND
Critical bug recall >= 90%
AND
False findings <= 1.0 / PR
AND
P95 latency <= 10 minutes
A system that fails a critical requirement shouldn't be rescued by a high score elsewhere.
Why "bug catch rate" is insufficient
This is the central point of the methodology.
Suppose Reviewer A catches 80% of known bugs.
That sounds great.
But now imagine:
Reviewer A
80% bug recall
25% precision
4 false findings / PR
and:
Reviewer B
70% bug recall
75% precision
0.6 false findings / PR
Which one is better?
There isn't enough information to answer.
It depends on:
Bug severity
Developer trust
Review volume
Latency
Cost
Acceptance rate
Human review process
Risk tolerance
This is why benchmark reports should never lead with a single "X% of bugs caught" number and stop there.
At minimum, publish the underlying dataset characteristics, ground-truth methodology, precision, severity breakdown, and false-positive behavior.
Make the benchmark reproducible
A credible benchmark should be something another engineering team can understand and, where possible, reproduce.
I'd publish:
Dataset specification
Number of PRs
Repository distribution
Language distribution
PR-size distribution
Real vs synthetic ratio
Bug categories
Severity distribution
Evaluation protocol
Available context
Model/system version
Tool configuration
Temperature or equivalent settings, if applicable
Number of evaluation runs
Date of evaluation
Annotation protocol
Ground-truth definition
Reviewer instructions
Number of annotators
Disagreement-resolution process
Metrics
Detection
Precision
False positives
Duplicates
Severity accuracy
Acceptance
Latency
Time saved
Raw results
Where security and privacy constraints permit, publish anonymized per-case results rather than only aggregate scores.
That makes the benchmark much harder to game.
Repeat the benchmark
AI systems change.
Models get updated. Prompts change. Repository tooling evolves. Vendors modify retrieval systems and analysis pipelines.
Your codebase changes too.
So treat the benchmark as a regression suite.
Keep a stable golden dataset and add new cases periodically.
For example:
v1.0
300 PRs
v1.1
300 original PRs
+ 50 newly labeled production bugs
v1.2
300 original PRs
+ 100 cumulative new cases
This lets you answer questions such as:
Did recall improve?
Did false positives increase?
Did critical-bug detection regress?
Did latency change?
Did developer acceptance improve?
A benchmark that only runs once quickly becomes historical trivia.
A benchmark that runs continuously becomes engineering infrastructure.
A practical evaluation report
If I were reviewing an AI code-review system for production deployment, I'd want the final report to look something like this:
| Dimension | Result | Notes |
|---|---|---|
| Known bugs | 118/150 | 78.7% recall |
| Critical bugs | 19/20 | 95% recall |
| Precision | 72% | 1.1 false findings/PR |
| Severity accuracy | 81% | Strong on critical/high |
| Duplicate rate | 6% | Mostly overlapping comments |
| Developer acceptance | 69% | Blinded evaluation |
| P95 review latency | 8.4 min | Large PRs slower |
| Estimated time saved | 14% | Pilot estimate |
| Reproducibility | Yes | Fixed dataset/protocol |
Then include the dataset, protocol, known limitations, and raw case-level results where they can safely be shared.
That report is considerably more useful than:
"Our AI catches 87% of bugs."
The benchmark should be open to criticism
There is no perfect benchmark for AI code review.
Any methodology introduces choices:
Which bugs count?
Which repositories?
Which languages?
What context is allowed?
Who labels ground truth?
How are duplicate findings handled?
How should severity be scored?
How much weight should developer acceptance receive?
How do we estimate time saved?
Those choices can materially affect the result.
That's not a reason to avoid benchmarking.
It's a reason to make the methodology explicit.
The most useful evaluation ecosystem would be one where engineering teams publish not only scores, but also datasets, protocols, annotations, limitations, and failed experiments.
That would move the conversation from vendor marketing toward evidence.
The goal isn't to prove that one AI reviewer is universally better than another.
It's to establish a measurement discipline where claims can be tested, reproduced, challenged, and improved.
If we're going to trust AI systems with an increasingly important part of software quality, the evaluation process should be held to the same engineering standards we expect from the systems themselves.
What would you add, remove, or change in this methodology? I'd especially like to hear how you would handle ground-truth labeling, synthetic bugs, developer time saved, and comparisons across very different repositories.
About the author
- Codzee is building the future of AI-powered code reviews for modern engineering teams. We help developers review pull requests faster, improve code quality, and ship software with confidence. Codzee analyzes your code, highlights potential issues, suggests meaningful improvements, and delivers actionable feedback that fits your team's coding standards and workflow. Our mission is simple: make code reviews faster, smarter, and more collaborative without disrupting how developers work. Whether you're a startup shipping your first product or a growing engineering organization, Codzee helps reduce review bottlenecks, maintain consistent code quality, and improve developer productivity. At Codzee, we believe AI should support developers, not replace them. That's why we're building AI Code Review That Respects Your Team.
Comments
Join the discussionNo comments yet. Be the first to comment.




