Tinker AI
Read reviews
4 min read Editor

Every team I’ve worked with that adopts AI tools experiences the same phenomenon a few months in. Code is shipping faster. CI passes. Code review is happy. Then bugs start appearing in production that have a specific shape: they’re not obvious bugs. They’re subtle wrongs that look right.

I’ve come to call these “AI quiet bugs.” They have specific characteristics that distinguish them from the bugs that humans tend to introduce.

What “quiet” means

The bug runs. The tests pass. The code review caught nothing. The code did exactly what someone asked it to do, syntactically. But the result is wrong.

Examples I’ve seen recently:

Wrong default behavior. A function asked to handle “empty input” defaults to returning []. The business logic actually wanted null to indicate “no result computed.” The function silently returns “computed empty list” when “no result yet” was the right answer. Months later, downstream code mistakenly treats “still loading” as “definitely empty” and ships the wrong UI to users.

Off-by-one in pagination. A pagination implementation has the right structure but loads page N when it should load page N-1. Tests pass because they happen to test page 0 (where N and N-1 produce the same result). Pagination starts misbehaving in production for users on page 2+.

Subtle race condition. Async code that “works” because the race rarely fires in normal load. Production at scale fires the race. By the time it’s noticed, customer data has been corrupted in 0.1% of cases.

Wrong unit. A function takes “duration” as a parameter. The AI-generated code treats it as seconds; the upstream code passes milliseconds. Conversion is consistent within the new code but wrong relative to the system. A 3-second timeout becomes 3-millisecond.

Sensible defaults that aren’t your defaults. AI-generated code uses Bcrypt with cost factor 10. Your security policy is cost factor 12. The code passes review because cost 10 is reasonable in general. Your security review fails six months later when audit notices.

Why these are AI-shaped

Each of these has a pattern: the code is “right” by some general standard, “wrong” by your specific standard. The AI’s training has many examples of “what’s reasonable in general.” It has few examples of “what’s reasonable for your specific business.”

Without explicit guidance, the AI defaults to general reasonableness. The general reasonableness is usually fine. Sometimes it isn’t.

A human writing the same code might:

  • Ask “should empty input return null or []?”
  • Notice that pagination test cases don’t cover N>1
  • Worry about the race condition because they’ve been bitten before
  • Verify the unit (or notice and ask)
  • Reference your security policy explicitly

The AI doesn’t ask these questions reliably. It picks a plausible answer and moves on.

Why review doesn’t catch them

Code review is calibrated to catch human-shaped mistakes. Humans tend to make:

  • Logic errors that don’t compile or fail tests
  • Style violations
  • Inconsistencies with surrounding code
  • Forgotten edge cases (where “forgotten” means visibly missing)

AI quiet bugs don’t have these characteristics. The code:

  • Compiles, types check, tests pass
  • Style is fine (matches AI’s training, often matches your conventions if rules are good)
  • Consistent with surrounding code (the AI saw the surrounding code)
  • Edge cases are handled (just maybe handled wrongly)

The visual signals reviewers use to flag suspicious code don’t trigger. The bug looks like working code.

What review needs to add

To catch quiet bugs, review needs to add specific questions:

“Are the defaults what we want?” Don’t assume reasonable defaults match yours. Verify. For functions with default behavior, check whether the default matches business intent.

“What’s tested vs. assumed?” Tests pass for tested cases. The untested cases are where bugs hide. Specifically check whether tests cover the case the AI most likely got wrong.

“What units, ranges, and shapes?” AI doesn’t always preserve units across refactors. Verify. Especially for numbers that have implicit units (seconds, dollars, percentages).

“What about concurrency?” Many quiet bugs are race conditions. Look for shared state, async access, missing locks.

“Does this match our specific policies?” Security, performance, data-handling policies. AI doesn’t know them unless told. Verify against them explicitly.

These checks add review time. They’re necessary if you’re shipping AI-assisted code at scale.

What teams should do

A few practical adjustments I’ve seen work:

Codify business defaults explicitly. “When a function returns ‘no data,’ use null, not [].” Document this. Reference in AI tool rules. The AI follows; review confirms.

Increase test coverage requirements. Quiet bugs hide in untested cases. Coverage isn’t a perfect signal but high coverage reduces the surface where bugs hide.

Add unit-aware types. TypeScript can express “this is seconds, that is milliseconds” via branded types. If your codebase has unit confusion potential, codify the units. AI follows the types.

Assume AI defaults are wrong and verify. Don’t trust “the AI generated reasonable defaults.” Check what the defaults are.

Run integration tests, not just units. Unit tests pass for AI-generated code more easily than integration tests do. The unit tests cover the AI’s view; integration tests cover the system’s view.

Slow down on critical paths. Authentication, payments, security checks — these don’t deserve AI-speed review. Treat AI-generated code in critical paths with extra skepticism.

A specific incident

A real example from a few months ago. A team I worked with had an authentication bug that shipped to production. The bug: AI-generated code was checking user.role === 'admin' instead of user.permissions.includes('admin'). The system had moved from role-based to permission-based auth six months earlier; old role checks would always pass for admin users, who happened to also have many permissions.

The new code:

  • Compiled
  • Type-checked (both shapes existed in the legacy types)
  • Passed tests (admin users are usually in test fixtures)
  • Passed review (looked like reasonable auth code)

In production, regular users got admin access. Discovered after a customer reported they could see another customer’s data.

The fix was trivial. The blast radius was 3 weeks of the bug being live.

The cause: the AI saw both patterns in the codebase, picked the older one for new code. The reviewer didn’t catch it because the older pattern was still in the codebase elsewhere; the inconsistency wasn’t an obvious red flag.

What this teaches

Quiet bugs are a category. They’re not a one-off. They’re the predictable result of:

  • AI tools with general reasonableness defaults
  • Codebases with mixed legacy and current patterns
  • Reviews calibrated to catch human-shaped mistakes
  • Test coverage that’s good but not perfect

Each of these is normal. The combination produces a class of bug that didn’t exist (in the same form) before AI tools.

The mitigations exist. The discipline of applying them is the gap. Most teams don’t realize they need new review questions until they’ve shipped a quiet bug.

What I’d watch for in your codebase

If you’re using AI tools and haven’t seen quiet bugs yet, you’re either:

  • Lucky
  • Not yet at scale where they manifest
  • Catching them via tests/review without realizing what category they’re in
  • About to encounter them

For the first three cases, no immediate action needed but stay alert. For the fourth case, you’ll learn the lesson the hard way.

Some signals that quiet bugs are accumulating:

  • Production incidents whose cause is “we had a wrong assumption baked in”
  • Customer reports of “this used to work differently”
  • Audits finding policy violations you don’t remember introducing
  • Test coverage increasing without bug reports decreasing

If you see these signals, audit recent AI-assisted code. Look for the patterns above.

Closing

AI tools are genuinely useful. They also introduce new failure modes that the existing engineering practices don’t fully catch. Adapting practices to the new failure modes is part of mature AI tool adoption.

Teams that figure this out early avoid expensive incidents. Teams that don’t figure it out end up debugging quiet bugs in production at high cost. The difference between the two trajectories is awareness — knowing this category exists and adjusting accordingly.

If your team uses AI tools and hasn’t talked about quiet bugs, it’s worth a discussion. The conversation alone changes how engineers review code; the changes in review behavior catch bugs that would otherwise ship.

Awareness is most of the cure. The rest is discipline.