Every few months, a new survey comes out claiming developers using AI tools are 30-70% more productive than developers who don’t. The numbers are striking. They’re also methodologically suspect in specific, predictable ways. Here’s a calmer reading of what the data actually shows.
The typical survey design
Most “AI productivity” surveys work like this:
- Recruit developers (often through GitHub, the AI tool vendors, or developer-focused mailing lists)
- Ask them: “Do you use AI coding tools?” Yes/No
- Ask both groups: “How productive do you feel?”
- Compute the difference; publish
This methodology produces inflated numbers for predictable reasons.
Bias 1: selection
The “Yes” group selected into AI tool adoption. The “No” group selected against it. Without random assignment, you can’t compare the groups directly.
Engineers who adopted AI tools are likely:
- More open to new tools generally
- More productive with experimental approaches
- More comfortable with the friction of learning new tooling
The “No” group, conversely, might include:
- Engineers in environments that prohibit AI tools (compliance, security)
- Engineers who tried AI tools and didn’t find them useful
- Engineers who haven’t gotten around to adopting
The “AI users are 50% more productive” finding partly captures “engineers who choose new tools are 50% more productive.” That’s a different claim than the headline suggests.
Bias 2: self-report
Self-reported productivity is famously unreliable. People who recently invested in a tool tend to report it works. People who didn’t invest tend to be neutral.
Engineers who paid for Cursor, set up Claude Code, learned Cline — they have reasons (financial, social, sunk-cost) to feel good about the tool. The “I’m 30% more productive” estimate is partly real productivity gain and partly cognitive reinforcement of the choice.
This isn’t dishonesty. It’s normal cognitive bias. We look for evidence that confirms our investments.
Bias 3: respondent population
Surveys distributed through GitHub Copilot’s mailing list, Cursor’s user community, Anthropic’s developer audience — these populations are pre-selected for AI adoption. The “developers using AI tools” are over-represented; the “developers not using AI tools” are under-represented.
Even when the survey explicitly recruits both groups, the response rate biases. AI adopters are excited; they fill out the survey. Non-adopters are indifferent; they skip it.
The published comparison ends up being “enthusiastic AI adopters vs. lukewarm non-adopters.” That’s not the comparison most readers think they’re seeing.
Bias 4: novelty
Many surveys run within 6-12 months of an AI tool’s launch. Engineers in the survey are still in the early-adoption phase, where productivity gains are exaggerated by:
- The novelty effect (new tools feel productive)
- The self-selection of motivated early adopters
- The lack of accumulated wisdom about failure modes
Surveys two years post-launch tend to show more modest numbers. Engineers who’ve used Copilot for three years report smaller productivity gains than engineers who started six months ago. The novelty effect dissipates.
What rigorous studies find
A few studies have used better methodology — controlled experiments where engineers do similar tasks with and without AI assistance:
- The original GitHub Copilot study (Mar 2023) showed ~55% faster on a specific task. Methodology: real but narrow task.
- A METR study (early 2025) found minimal speedup on real-world coding work, despite engineers feeling 20% faster.
- Several smaller studies have shown the gain varies wildly by task type — large gains on greenfield, small gains on legacy.
The consistent finding: real productivity gain exists but is smaller than self-report. The discrepancy is the bias the simple surveys are missing.
My honest estimate
Based on the research and my own observations, the actual productivity gain from AI tools is roughly:
- Greenfield work, mainstream stack: 20-40% gain. Real and meaningful.
- Legacy work, well-known codebase: 5-15% gain. Modest.
- Niche language or framework: 5-25%. Highly variable.
- Embedded, real-time, security-critical: 0-10%. Sometimes a small loss.
These numbers are nowhere near the 50-70% the marketing surveys suggest. They’re also not zero. AI tools genuinely help for a real range of tasks.
Why the inflated numbers persist
Several reasons the inflated numbers continue circulating:
Publication bias. Studies finding large effects get press; studies finding modest effects don’t. The visible numbers skew high.
Vendor incentives. Tool vendors have incentive to publicize favorable studies. Anthropic, GitHub, Cursor — all have published or sponsored surveys with high productivity numbers.
Engineer pride. Engineers who use AI tools feel good about it. Saying “I’m 50% more productive” is more impressive than “I’m 15% more productive.” The narrative reinforces.
Novelty fades slowly. Several years into AI tool adoption, novelty effects are still present in some segments. The numbers haven’t fully settled.
What teams should track instead
For a team trying to assess AI tool ROI, the surveys are nearly useless. Better measurements:
Cycle time. From PR open to merge. If your AI tools are working, cycle time should drop modestly.
Defect rate. Bugs reported per merge. Should be flat or slightly down. If it goes up, AI tools are introducing more bugs than they’re saving.
Reviewer load. Time per reviewer. Should be flat or slightly down. If it goes up, AI tools are shifting work to reviewers.
Engineer satisfaction. Self-report still useful directionally. “On a scale of 1-10, how much does this tool help you?” If trending up over a year, the tool is working.
Output quality. Customer-reported issues, support tickets, escalations. The metrics that actually matter for the business.
These are harder to measure than survey responses. They’re also more honest. A team that sees “cycle time down 12%, defect rate flat, reviewer load up 5%” has a more accurate picture than one trusting a “users report 50% gain” survey.
The honest pitch
If I were marketing an AI coding tool, the honest pitch would be:
“Most engineers are 15-30% faster on their day-to-day work after a few months of practice. The gains are larger on greenfield projects in mainstream stacks, smaller on legacy or niche work. The tools also introduce specific failure modes that require attention. Net effect: meaningful productivity improvement that compounds over time, not a transformation.”
This pitch wouldn’t go viral. It would be more accurate.
Closing observation
The “AI tools change everything” narrative is a marketing artifact. The actual story is more modest and more interesting. AI tools are real productivity improvements that work better in some contexts than others, that take time to learn, that have specific failure modes.
Engineers who calibrate to the real picture make better decisions about adoption, training, and team workflows than engineers who calibrate to the marketing numbers. The expectations are appropriate; the disappointment is avoided; the actual value is captured.
The next time you see a survey claiming 60% productivity gains, treat it the way you’d treat any marketing material. The signal is “this tool is helpful.” The specific numbers are noise.