Skip to main content

Can AI Code Review Catch 18 Defects ? We Compared Codzee, Greptile, CodeRabbit, and CodeAnt

Updated 7 min readMarkdown
Can AI Code Review Catch 18 Defects ? We Compared Codzee, Greptile, CodeRabbit, and CodeAnt

A pull request says its tokens are secure. Its sessions expire correctly. Its permission model fails safely.

But none of those claims is true.

This is the kind of situation where AI code review needs to do more than summarize a diff. It needs to verify what the code actually does.

We tested four AI code review tools against a deliberately vulnerable pull request to find out how they handled misleading security claims, additional vulnerabilities, and defective tests.

The results revealed both strengths and weaknesses, including some in our own product, Codzee.

1. The experiment: One pull request, four AI code review tools

For the tenth round of our benchmark, we compared:

  • Codzee

  • Greptile

  • CodeRabbit

  • CodeAnt

The test repository contained 18 planted security and logic defects across three source files. Four security claims in the pull request description directly contradicted the implementation.

We also introduced six defects in the test file, which we scored separately.

Each tool reviewed the same head commit. We collected findings through the GitHub REST API and compared them against an answer key prepared before the branch was pushed.

The repository was a sandbox designed specifically for evaluating code review tools. No real credentials, systems, or user data were involved.

2. AI code review benchmark results

Tool Source-code defects detected
Codzee 17/18
Greptile 14/18
CodeRabbit 12/18
CodeAnt 10/18

Codzee achieved the highest score in this round. However, the defect categories reveal more about how the tools performed.

Category Codzee Greptile CodeRabbit CodeAnt
Four denied security claims 4/4 4/4 3/4 4/4
Ten additional security defects 10/10 7/10 5/10 4/10
Four logic defects 3/4 3/4 4/4 2/4
Six test defects, scored separately 6/6 0/6 0/6 0/6

The main result was not simply that Codzee detected more defects. It was where the differences appeared.

Every tool detected at least three of the four security defects that the pull request description claimed were already handled. The largest separation came from the additional security defects that the author never mentioned.

3. Four security claims that the code contradicted

Forged tokens

The description claimed that tokens were signed and verified using HMAC-SHA256.

The implementation used Base64 encoding:

Buffer.from(payload).toString("base64")

Encoding does not provide a cryptographic signature. Without signature verification, an attacker could construct a token claiming an administrator role.

All four tools detected this defect.

Broken session expiration

The description said expired sessions were rejected.

However, the code compared expiresAt with issuedAt instead of checking the current time. The expiration check did not enforce the intended session lifetime.

Again, all four tools detected the defect.

Fail-open permissions

The stated permission model was deny-by-default, but unknown roles received fallback read and write permissions.

A role typo could therefore grant access instead of rejecting the request.

All four tools caught this issue as well.

Insecure secret comparison

The implementation compared secrets using ordinary string equality:

provided === SECRET

That did not satisfy the claimed timing-safe comparison property. CodeRabbit was the only tool that missed this defect.

4. The difference between finding a defect and understanding the requirement

Three competing tools explicitly connected their findings to the incorrect claims in the pull request description.

Codzee detected all four defects but did not explicitly identify the contradiction between the author's claims and the implementation.

That distinction matters in security reviews.

A finding that says a comparison is not timing-safe identifies the technical problem. A finding that also references the security property promised by the author explains why the implementation fails to meet its own requirements.

The latter provides useful context for the reviewer and author.

There was another shared gap: none of the tools explicitly recommended correcting the misleading pull request description before merging.

Fixing the code without correcting its documentation can leave future engineers with an inaccurate understanding of the system's security guarantees.

5. Where Codzee found additional security defects

The ten additional security defects produced the largest separation in the benchmark.

Codzee detected all ten, compared with seven for Greptile, five for CodeRabbit, and four for CodeAnt.

The issues included insecure authorization behavior, missing ownership checks, stale permissions, weak session-ID randomness, a hardcoded fallback secret, and user enumeration through login errors.

Three findings were unique to Codzee:

  • Session identifiers generated using Math.random() rather than a cryptographically secure random number generator.

  • A hardcoded fallback secret that could allow a misconfigured deployment to start with a known key.

  • Login error messages that revealed whether a username existed.

These findings illustrate why an AI code review evaluation should include vulnerabilities that are not explicitly described in the pull request.

6. The test-file result deserves attention

Codzee found all six planted test defects. The other three tools found none.

Two test cases were particularly concerning because they treated insecure behavior as expected behavior: constructing a session from an x-user-id header and allowing access to another user's booking.

This is a different class of problem from an ordinary missed bug. Tests can actively reinforce an insecure implementation when they assert the wrong behavior.

The result suggests that test-file analysis deserves its own place in AI code review evaluations. However, it is limited to the six test defects included in this particular experiment.

7. What Codzee missed, and what we need to improve

Codzee's score was 17/18, not 18/18.

The missed defect was in logout(). The function did not invalidate the token, so the session could remain valid after logout.

The benchmark also exposed three product-level gaps.

Finding visibility: Only 10 of Codzee's 29 findings appeared as inline comments. The other 19 were placed in a collapsed block, making some findings easier to overlook.

GitHub enforcement: Codzee reported ten Critical security findings but returned a neutral check conclusion. Its legacy commit status simultaneously reported success. A neutral check may allow a merge to proceed even when critical issues have been identified.

Pull request context: Codzee did not explicitly reference the four false security claims in the description, even though three competing tools connected at least some findings to those claims.

Our next priorities are to improve critical-finding enforcement, increase the visibility of important findings, and include the pull request title and description in the review context.

8. What engineering teams should evaluate

This benchmark involved one deliberately constructed sandbox pull request. It does not establish how these tools will perform across all production repositories, programming languages, or defect classes.

But it provides a useful evaluation framework for engineering teams comparing AI code review tools.

Detection accuracy: Does the tool find defects that are not mentioned in the pull request?

Requirement verification: Does it compare the implementation against the security guarantees described by the author?

Test analysis: Can it recognize tests that validate vulnerable behavior?

Finding visibility: Are important issues available where developers will actually see them?

Merge enforcement: Can a Critical finding block a merge according to the team's security policy?

A strong review process needs to consider all five. Detection is essential, but the value of a finding also depends on whether developers can understand it and whether the workflow responds appropriately.

Conclusion

Codzee detected 17 of 18 planted source-code defects in this round, including all ten additional security defects and all six separately evaluated test defects.

That is a strong result for this experiment, but the benchmark also revealed weaknesses in how Codzee communicates findings and handles critical GitHub check outcomes.

We believe publishing both sides of the result is important. AI code review should be evaluated through transparent experiments, not just product claims or aggregate scores.

At Codzee, our focus is on helping engineering teams identify defects, understand security risks, and make better-informed decisions about the code they merge.

Explore the product at https://www.codzee.io.

For engineering managers and developers: What would make you trust an AI code review tool more: higher detection accuracy, stronger reasoning about requirements, or enforceable security checks?.

1 view

About the author

  • Codzee is building the future of AI-powered code reviews for modern engineering teams. We help developers review pull requests faster, improve code quality, and ship software with confidence. Codzee analyzes your code, highlights potential issues, suggests meaningful improvements, and delivers actionable feedback that fits your team's coding standards and workflow. Our mission is simple: make code reviews faster, smarter, and more collaborative without disrupting how developers work. Whether you're a startup shipping your first product or a growing engineering organization, Codzee helps reduce review bottlenecks, maintain consistent code quality, and improve developer productivity. At Codzee, we believe AI should support developers, not replace them. That's why we're building AI Code Review That Respects Your Team.

No comments yet. Be the first to comment.

More from this blog