What we learned from 2,000 reviewed pull requests

Eblan crossed 2,000 reviewed pull requests this week. That's a small number by the standards of a mature product, but it's the first dataset we have that's big enough to learn from — so we spent a few evenings digging through it. Here's what we found, including a couple of things that surprised us.

What Eblan actually catches

We sampled 400 comments and asked: would a competent human reviewer have left this? (When we weren't sure, we asked the beta team that received the comment.)

31% flagged an issue a human confirmed as real — a bug, a security concern, or a performance problem worth fixing.
44% were "good hygiene" comments — missing tests, unclear naming, error paths nobody had thought about. Not urgent, but welcome.
25% were things humans would probably have skipped. We're working on that last bucket.

For a tool whose whole pitch is "worth reading," that last number is the one we watch most. A year ago, a 25% miss rate would have been a miracle; today it's our roadmap.

The most common catch surprised us

We expected race conditions and N+1 queries to top the list. They didn't. The single most common real issue Eblan catches is unhandled failure paths — code that works perfectly when everything works: a fetch whose error is swallowed, a lock that's never released on the early-return path, a background job that dies silently on a malformed payload.

In hindsight it makes sense. Humans review the happy path because it's what the diff shows. A model that reads the surrounding code sees the unhappy paths too — and unlike a linter, it can tell which ones actually matter.

Small teams review differently

The beta skews toward solo founders and teams of two to five, and their usage pattern is distinct: they don't use Eblan instead of human review, they use it instead of having no review at all. The median PR for a solo founder gets one human look (their own) and one Eblan review. For teams of five or more, Eblan usually comments first, and humans read the thread after.

One solo founder told us the thing that keeps making our week: "It's the first tool that makes merging my own code feel responsible instead of reckless."

What we're fixing

Three things, in priority order:

Fewer low-value comments. We've retrained our filtering on the "humans would skip this" bucket, and early numbers suggest we can cut it roughly in half without losing real catches.
Better suggestions on large diffs. When a PR touches thirty files, Eblan's comments get more generic. We're changing how context is prioritized for big changesets.
Faster reviews on monorepos. Context gathering is our latency bottleneck on very large repos. Monorepo-aware fetching is in progress (see the changelog).

If you're in the beta: thank you. Every "this caught a real bug" email and every "meh, this comment was noise" reply feeds straight back into the product. If you're not — the waitlist is open, and we're onboarding weekly.

— The Eblan team