Skip to main content

AI Code Review Benchmark: Codzee vs. Greptile, CodeAnt, and CodeRabbit

Updated 8 min readMarkdown
AI Code Review Benchmark: Codzee vs. Greptile, CodeAnt, and CodeRabbit

An 11-defect refactoring experiment reveals how AI reviewers handle broken fixes, behavioural regressions, and deleted tests.

Introduction

A code review can look successful while leaving a serious defect behind.

A developer refactors an existing module, fixes several reported issues, and simplifies the implementation. The resulting diff looks cleaner, but some previously correct behaviour has changed. A test disappears, and an edge case slips through.

This raises an important question for teams adopting AI-assisted development:

Can AI code review tools reliably identify defects introduced when existing code is modified?

To investigate, we compared Codzee, Greptile, CodeAnt, and CodeRabbit against the same controlled pull request.

In this eleventh benchmark round, Codzee detected nine of eleven planted defects. The detailed findings also exposed limitations shared by all four tools.

This article explains the experimental setup, detection results, representative defects, and implications for engineering teams.

Experimental design

The benchmark evaluated the head commit of GitHub pull request #12.

The change modified two existing files and contained:

  • 46 insertions

  • 34 deletions

  • 11 planted defects

  • Two deliberately correct refactors used as false-positive checks

The eleven defects consisted of three claimed fixes that remained incomplete or incorrect and eight regressions introduced by restructuring existing code.

We collected findings using the GitHub REST API and scored them against an answer key written before the branch was pushed.

Beyond total detection, we evaluated four additional dimensions:

  1. Detection of previously reported problems that remained unresolved.

  2. Detection of behavioural regressions.

  3. Recognition of deleted tests and missing coverage.

  4. Inline comment placement, severity assessment, and reasoning about prior implementation behaviour.

The objective was to compare meaningful defect detection rather than raw comment volume.

Results: Round Eleven

Codzee detected nine of eleven planted defects, giving it the highest score in this round.

Greptile detected eight, CodeAnt detected six, and CodeRabbit detected five.

Detection breakdown

Codzee

  • Claimed fixes still broken: 2/3

  • New regressions: 7/8

  • Total: 9/11

Greptile

  • Claimed fixes still broken: 1/3

  • New regressions: 7/8

  • Total: 8/11

CodeAnt

  • Claimed fixes still broken: 2/3

  • New regressions: 4/8

  • Total: 6/11

CodeRabbit

  • Claimed fixes still broken: 1/3

  • New regressions: 4/8

  • Total: 5/11

The main difference between Codzee and Greptile was not regression detection: both identified seven of the eight new regressions. Codzee detected one additional unresolved claimed fix.

These results apply to this specific test scenario. They should not be interpreted as a general ranking across all codebases, languages, or workloads.

Case study 1: Validation logic and negative inputs

One of the most significant missed defects involved a loyalty-points redemption function.

The intended change was to prevent redemption beyond an account's available balance. The implementation added a guard checking whether the requested points exceeded the balance.

However, the condition did not reject negative redemption amounts.

For example:

redeem(acct, -100)

could pass the guard and credit points to the account rather than rejecting the request.

All four tools missed this defect.

The failure illustrates a gap between recognizing the intent of a fix and verifying its complete input domain. The guard addressed the specific over-redemption case but did not establish that the operation was valid for negative inputs.

For engineering teams, this reinforces the importance of boundary-value tests, negative-input tests, and explicit invariants for financial or points-based operations.

AI review can help identify these issues, but it should not replace systematic input validation and automated testing.

Case study 2: Behavioural regressions during refactoring

Another defect involved a tier lookup disrupted by a sorting operation. Every positive balance could be assigned to the bronze tier.

All four tools identified this regression.

Other defects required more careful comparison with the previous implementation.

A function changed from returning a list to returning a generator. Another converted an integer-based monetary value into a floating-point representation. Additional regressions included a removed negative-spend guard and a new side effect in a function that previously did not have it.

These changes demonstrate why reviewing the final implementation alone can be insufficient.

A return-type change can break callers that rely on list semantics. A representation change can introduce precision or cross-language consistency problems. Removing a guard can alter behaviour even when the main execution path still appears correct.

A useful refactoring review should compare old and new behaviour, including:

  • Function signatures and return types.

  • Input validation and boundary conditions.

  • Side effects and mutation.

  • Data representations.

  • Assumptions made by callers and related implementations.

Case study 3: Deleted tests and missing coverage

The pull request removed tests, including an assertion related to tier behaviour.

The deleted assertion was itself incorrect, so removing it was defensible. The problem was that no replacement test covered the intended behaviour.

Codzee was the only tool to flag the deletion and explain the missing coverage.

This finding highlights an important distinction: detecting a removed test is not enough. The reviewer must understand which behaviour the test was meant to protect and whether that behaviour remains covered.

Test changes deserve first-class review because they can weaken the safety net that protects future modifications.

A shared blind spot: The mutable default argument

The benchmark also included a mutable module-level list exposed as a default argument. None of the four tools detected it.

The research identified this as a pattern covered by Ruff rule B006.

This result illustrates why complementary tooling remains valuable. Linters and static analysis can detect certain classes of problems consistently, while AI reviewers can contribute contextual reasoning about changes and their consequences.

The strongest engineering workflow uses both rather than assuming either is sufficient on its own.

Review volume and inline findings

The research found that review volume fell by 39% across all four tools compared with the previous round.

This suggests that the refactor's shape affected how much each tool reported. The defects that survived were often those requiring comparison with previous behaviour or verification of a claimed fix.

Codzee produced eight findings in this round, and all eight were posted inline.

Across the previous ten rounds, Codzee's inline counts were 5, 6, 0, 3, 2, 10, 10, 10, 10, and 10. The latest result supports the benchmark team's hypothesis that the earlier results reflected a ten-comment ceiling rather than unreliable code anchoring.

The benchmark also identified an issue with check status: a critical finding could coexist with a successful GitHub check. The research recommends configurable failure behaviour when critical findings are present.

These are important distinctions. Finding a defect, placing the finding where developers can act on it, and communicating the merge risk are separate responsibilities.

Results across eleven rounds

The team's running tally reported six rounds led or tied for Codzee, six for Greptile, one for CodeAnt, and one for CodeRabbit. Codzee had won the last five rounds outright.

The benchmark series covered multiple scenarios, including payment logic, authentication, asynchronous behaviour, test quality, and Python/TypeScript parity.

Because the scenarios and defect counts varied, the round tally should be treated as a historical summary rather than a normalized aggregate score. The strongest conclusions remain those supported by the detailed results from each individual round.

Implications for engineering teams

This benchmark offers several practical recommendations for evaluating AI code review:

Test changes, not just final code. Include regressions that become visible only when comparing old and new implementations.

Challenge claimed fixes. Verify the intended behaviour and explore inputs that could bypass the new logic.

Review test deletions. Determine whether removed assertions were incorrect, redundant, or still protecting important behaviour.

Combine AI review with static analysis. Use complementary tools to catch patterns that contextual review may miss.

Make severity actionable. Ensure that critical findings are clearly communicated and that GitHub check status reflects the team's merge policy.

No single benchmark establishes a universal winner. Teams should evaluate tools against representative changes and failure modes from their own repositories.

How Codzee uses these findings

At Codzee, we use controlled benchmark rounds to measure defect detection and identify opportunities to improve the product.

Round Eleven highlighted strengths in regression detection and test-file review, while also exposing areas for improvement in negative-input reasoning and critical-finding check status.

The objective is to make code review more useful for developers: fewer missed regressions, clearer findings, and better context around the changes being proposed.

If you're evaluating AI code review for your team, explore Codzee and assess it against the kinds of defects your codebase actually encounters.

The goal is not simply to produce more comments. It is to help engineering teams identify meaningful problems before those problems reach production.


Methodology note

Round Eleven evaluated the head commit of GitHub pull request #12, containing 46 insertions and 34 deletions across two existing files. Eleven planted defects were scored against an answer key prepared before the branch was pushed. The findings and scores reported here are specific to this controlled benchmark and should not be interpreted as a universal comparison of the products.

1 view

About the author

  • Codzee is building the future of AI-powered code reviews for modern engineering teams. We help developers review pull requests faster, improve code quality, and ship software with confidence. Codzee analyzes your code, highlights potential issues, suggests meaningful improvements, and delivers actionable feedback that fits your team's coding standards and workflow. Our mission is simple: make code reviews faster, smarter, and more collaborative without disrupting how developers work. Whether you're a startup shipping your first product or a growing engineering organization, Codzee helps reduce review bottlenecks, maintain consistent code quality, and improve developer productivity. At Codzee, we believe AI should support developers, not replace them. That's why we're building AI Code Review That Respects Your Team.

No comments yet. Be the first to comment.

More from this blog

AI Code Review Benchmark: Codzee vs. Greptile, CodeAnt, and CodeRabbit