AI Coding Tools and Productivity: What Research Found

AI coding tools can make a developer feel faster while making a particular task take longer. They can increase the amount of code produced while also increasing review work, duplication, or security findings. Those statements are not necessarily contradictory. They describe different outcomes, measured at different levels.

That distinction is the most useful way to read the research on AI coding tools and productivity. Controlled experiments measure task completion under defined conditions. Repository analyses observe patterns in code changes. Security evaluations test whether generated snippets choose safe implementations. Surveys capture adoption, trust, and perceived impact. None of those measurements is a substitute for all the others.

The evidence supports neither a universal productivity boost nor a universal slowdown. It supports a narrower conclusion: AI changes the distribution of work. It can reduce effort in some tasks and contexts, while shifting cost into verification, integration, review, and maintenance. Teams need to measure those stages separately.

Why developer productivity is difficult to measure#

“Productivity” sounds like one metric, but software development produces several outputs. A developer might finish a ticket sooner, write more code, resolve more tasks, reduce time spent searching, or improve a product outcome. Those are related, but they are not interchangeable.

A randomized task experiment can compare time or completion between AI-assisted and unassisted conditions, but a short task may not represent work in a large codebase or include downstream review costs. Field experiments are more realistic but vary by task, tool, and team. Surveys capture adoption and sentiment, not causal task speed. Repository analyses reveal code-change patterns, but trends do not establish what caused them.

A useful reading rule is to ask four questions before accepting a headline:

Who or what was measured? A developer, a task, a commit, a repository, a generated function, or a team?
What counted as success? Less elapsed time, more completed work, accepted code, fewer defects, or a positive self-report?
What was the comparison? No AI, a previous year, another team, a different model, or a vendor-defined baseline?
What remains outside the measurement? Review, debugging, security, maintainability, onboarding, or the value of the shipped product?

Without those details, “AI productivity” is too broad to be a useful conclusion.

Controlled studies: faster in some settings, slower in others#

Controlled studies provide the clearest task-level comparisons, but their results depend heavily on the task and participants.

GitHub Copilot: a narrow task completed faster#

In a 2023 controlled experiment, 95 professional developers were randomly split between implementing a JavaScript HTTP server with or without GitHub Copilot. The Copilot group completed the task 55.8% faster on average. This is evidence for acceleration on a bounded task—not a universal speedup. The study did not measure long-term maintenance or team delivery, and tested an earlier version of the tool. Read the study.

METR: experienced maintainers took longer#

METR’s randomized controlled trial, published July 10, 2025, tested a different setting. Sixteen experienced open-source developers worked on 246 real issues in mature repositories they had contributed to for multiple years. The repositories averaged more than 22,000 stars and one million lines of code. Developers had moderate AI experience and an average of about five years of experience with the repositories.

Each issue was randomly assigned to an AI-allowed or AI-disallowed condition. Participants could choose their tools; in the AI condition they primarily used Cursor Pro with Claude 3.5 or 3.7 Sonnet. Tasks averaged about two hours. Participants recorded their screens and self-reported implementation time. METR reported that allowing AI increased task completion time by 19%. Before the tasks, developers expected a 24% time reduction; afterward, they still estimated that AI had made them 20% faster.

The result is striking because it contrasts perceived speed with measured time in a realistic, repository-specific setting. But METR explicitly cautioned against generalizing it to most developers or software work. The sample was small and highly experienced, and the tested tools represented the early-2025 frontier. METR also listed possible improvements in prompting, scaffolding, and repository-specific context as reasons the result might not transfer to other setups. Read METR’s study and its limitations.

A February 2026 update described a later experiment with 57 developers, 143 repositories, and 800+ tasks. METR said selection effects made its estimates unreliable: some developers declined to work without AI, and 30%–50% said they withheld tasks they did not want to attempt without it. Raw estimates suggested possible speedups, but confidence intervals included no effect. The update does not erase the 2025 result; it reinforces that it was time- and context-bounded. Read the update.

Three workplace field experiments: more tasks completed#

A study by Kevin Cui and co-authors combined three randomized field experiments at Microsoft, Accenture, and an anonymous Fortune 100 company. Across 4,867 developers given access to an AI code-completion assistant or not, the combined analysis found a 26.08% increase in completed tasks (standard error: 10.3%). Less experienced developers had higher adoption and larger gains.

The individual experiments were noisy and varied. Completed task count is not the same as time per task or long-term code quality. The working paper was revised in August 2025 and published in Management Science on February 27, 2026. It counters a blanket slowdown narrative, but covers code-completion assistants in three enterprise settings—not every agentic workflow. Read the study.

These studies disagree only if we treat task time, task count, and work context as the same outcome. A bounded JavaScript task, a mature open-source issue, and ordinary enterprise work create different opportunities for assistance and different costs of review.

Large codebase analyses: quality signals, not a causal verdict#

GitClear’s February 2025 report, AI Copilot Code Quality: Evaluating 2024’s Increased Defect Rate via Code Quality Metrics, analyzed 211 million meaningful changed lines from January 2020 through December 2024. Its data combined roughly two-thirds anonymized data from private corporations that opted in and one-third open-source projects, including projects associated with Google, Meta, and Microsoft.

GitClear classified code changes into operations such as added, moved, updated, copy/pasted, and churned lines. It reported that moved lines—used as a proxy for refactoring—fell from 24.8% of changed lines in 2021 to 9.5% in 2024. Copy/pasted lines rose from 8.4% to 12.3% over the same period. In 2024, copy/pasted lines exceeded moved lines for the first time in the report’s series. GitClear also found that commits containing a duplicated block of at least five contiguous lines rose from 0.45% in 2022 to 6.66% in 2024 in the commits it scanned.

These are maintainability signals, not direct measures of maintenance difficulty. Repeated code can create more places to update, but the time trend does not prove AI caused it or establish a randomized AI-versus-no-AI comparison. GitClear’s classification excludes generated files and other changes, includes comments, and does not separate test from production code. Its historical duplicate-block scan examined the largest 1,000 commits per repository for earlier years, which it says may bias those years toward finding more duplication. Treat the results as a reason to inspect local trends—not proof that AI-generated code is inherently duplicated. Read the report.

Security research: code that compiles can still be unsafe#

Security studies test a narrower question than productivity studies: when a model is asked to generate code, does it choose a secure implementation?

Veracode’s 2025 GenAI Code Security Report evaluated more than 100 language models using 80 code-completion tasks across Java, JavaScript, C#, and Python. The tasks covered four vulnerability classes: SQL injection, cross-site scripting, log injection, and use of broken or risky cryptographic algorithms. Each task supplied a short function with a missing implementation and a comment describing the desired behavior. Veracode ran static analysis against the generated code.

Across the tested models and tasks, Veracode reported an average security pass rate of approximately 55%—meaning 45% of generated completions contained one of the known security flaws being tested. Results varied sharply by vulnerability type: models performed much better on SQL injection and cryptographic choices than on cross-site scripting and log injection. The report found little meaningful improvement in security performance across model sizes or release dates in its tested set.

The limitation is central to the result. These were intentionally constrained function-completion tasks with minimal context and no security-specific prompting. Veracode did not check functional correctness; it checked compilation and the selected security properties using its own static analysis engine. The report therefore does not mean 45% of production AI code is vulnerable. It means the tested models often chose insecure implementations under these task conditions. The vendor’s own report discusses the trade-off and the narrowness of the test. Read the report.

Apiiro’s September 2025 report looked at a different level: repository and developer activity inside Fortune 50 enterprises. Using its proprietary Deep Code Analysis engine, Apiiro analyzed tens of thousands of repositories and several thousand developers using a variety of coding assistants. It reported that AI-assisted developers produced three to four times more commits but fewer, larger pull requests; the company also reported ten times as many security findings, with more than 10,000 new findings per month by June 2025 compared with December 2024.

Apiiro reported a mixed risk profile: syntax errors fell 76% and logic bugs fell by more than 60%, while privilege-escalation paths rose 322% and architectural design flaws rose 153%. It also reported that AI-assisted developers exposed certain Azure credentials nearly twice as often as non-AI peers.

Those are Apiiro’s findings from its own enterprise telemetry, not an independently replicated causal experiment. The report does not provide enough public detail to independently reconstruct the full sampling, attribution, or normalization behind every comparison. Its “security findings” include a broad set of risks and should not be read as a count of confirmed exploitable vulnerabilities. The company also sells application-security products, so its commercial position belongs in the reader’s assessment. The evidence is a warning about the possible security workload created by high-volume changes, not a universal vulnerability multiplier. Read Apiiro’s report.

Surveys: adoption and trust are not productivity measurements#

The 2025 Stack Overflow Developer Survey received 49,009 responses from 177 countries. It was fielded from May 29 to June 23, 2025, with respondents recruited primarily through Stack Overflow-owned channels. The survey reported that 84% of respondents used or planned to use AI tools in their development process, while 51% of professional developers said they used AI tools daily.

The same survey showed a trust gap: 46% distrusted AI output accuracy, compared with 33% who trusted it; 66% reported frustration with answers that were almost right, and 45% said debugging AI-generated code could take more time. These are self-reported sentiments, not causal productivity measures. Recruitment through Stack Overflow-owned channels also means the sample is not a probability sample of every developer worldwide. Read the survey and methodology.

DORA’s 2025 State of AI-assisted Software Development report used survey responses from nearly 5,000 technology professionals and more than 100 hours of qualitative data. The survey ran from June 13 to July 21, 2025. DORA reported that 90% of respondents used AI at work and more than 80% believed it increased their productivity. Its analysis associated greater AI adoption with higher software delivery throughput and product performance, but also with greater software delivery instability. DORA’s central interpretation was that AI amplifies the strengths and weaknesses of the surrounding organizational system.

This is organizational evidence, not a randomized trial of individual task speed. The reported relationships are associations, not proof that AI alone caused a team to improve or deteriorate. DORA moves the question beyond typing speed to whether increased output becomes a better product. Read DORA’s 2025 research overview.

Where the evidence agrees—and where it does not#

Across these sources, several patterns are consistent:

Perceived productivity is not measured productivity. METR’s 2025 participants believed AI helped even when their measured task time increased. Surveys capture perception; experiments can compare observed task outcomes.
Task and context matter. A narrow implementation task, a mature repository issue, and enterprise delivery work do not test the same capability. Results should not be averaged into one universal speedup.
More code is not the same as more value. Commit volume, lines changed, and completed tasks are output measures. They do not automatically account for review cost, defects, maintainability, or product outcomes.
Verification remains part of the work. Veracode and Apiiro identify security risks under different methods, while Stack Overflow respondents report trust and debugging concerns. DORA connects AI adoption to delivery instability as well as throughput.

The disagreements are also real. GitHub’s bounded task, Microsoft’s three field experiments, and DORA’s 2025 organizational analysis report positive productivity or throughput signals. METR’s 2025 RCT reports a slowdown for experienced maintainers working on their own repositories. GitClear’s code-change trends and Apiiro’s security telemetry identify possible downstream costs. These findings should not be collapsed into a vote. They differ in period, tools, participants, tasks, outcomes, and measurement units.

What engineering teams should measure#

For an engineering manager, the practical response is not to choose the most optimistic or pessimistic paper. It is to run a local evaluation that separates the stages where AI may save or add work.

Choose representative tasks. Include routine implementation, unfamiliar code, bug fixes, refactoring, tests, documentation, and cross-service changes. Do not benchmark only the tasks where the tool is strongest.
Record total cycle time. Measure from task start to accepted merge, including prompting, waiting, debugging, review, rework, and test failures. Keep active developer time separate from elapsed time when agents run asynchronously.
Measure accepted outcomes. Track completed and accepted tasks, review rounds, change size, reverts, escaped defects, and rework. Avoid treating lines of code or commits as productivity by themselves.
Inspect maintainability. Track duplication, churn, refactoring, dependency changes, and the age of code being modified. Compare these measures against your own baseline rather than assuming GitClear’s dataset describes your repository.
Add security checks. Review authentication and authorization boundaries, data handling, secrets, dependency provenance, and security-sensitive code paths. Static analysis and tests are useful controls, but neither replaces architectural review.
Segment the results. Compare by task type, developer experience, repository familiarity, tool/model, and workflow. A team-wide average can hide where assistance helps and where it creates rework.
Re-evaluate after tool changes. Model capabilities and agent workflows change quickly. A result from early 2025 is not a permanent property of AI-assisted development.

The most useful baseline is the team’s existing performance on comparable work. Run a small, time-bounded pilot, define success before starting, and keep the evaluation long enough to observe review and rework—not just initial code generation.

What I think: verification is the missing denominator#

My interpretation: the central mistake in many AI productivity claims is measuring the generation step as though it were the whole engineering task. Code can appear quickly, but it still has to be understood, tested, reviewed, secured, integrated, and maintained. If the measurement ends when code is produced, it misses the work that determines whether the output is useful.

That is why verification deserves to be measured as a first-class part of the workflow. It is also the connection to two related Letinix essays: Code Generation Is Fast. Verification Is Not. and The Competence Debt of Agentic Coding. The first explores the verification bottleneck; the second examines the risk of delegating work that the developer can no longer confidently explain or assess. Their URLs were not supplied with this assignment, so they are noted for internal linking rather than linked to guessed destinations.

This is an editorial judgment, not a result directly established by any one study. The research supports the need to distinguish output, time, quality, security, and trust. It does not yet provide one agreed metric that captures the full value of AI-assisted engineering across organizations.

What to do next#

If you want to use AI more deliberately, start with the AI Prompt Pack for Developers. Treat prompts as a way to improve task framing and context—not as a substitute for code review, tests, security analysis, or engineering judgment.

References#

Spotted an error? Tell us.How we research and label evidence: About Letinix.

Muhammad Azlaan Zubair

Software Engineer and Researcher, writing about AI, developer tooling and automation.

Website GitHub LinkedIn Instagram Threads X ORICID Google Scholar

AI Coding Tools and Productivity: What the Research Actually Found