Does AI actually make people more productive?

The Productivity Paradox: Big Study Gains, Disappointing Real-World Results

Here's the puzzle: in controlled studies, generative AI looks like a miracle. Across three empirical case studies, AI improved workers' throughput by 66% on average — with programming tasks up 126%, business writing up 59%, and customer support up 13.8%. All statistically significant, with quality improving alongside speed in two of the three studies.

Then you look at what happens outside the lab, and the picture darkens considerably.

A large-scale study of 700+ software firms found that AI coding agents did exactly what they promised on the surface: code volume rose 30%, commits rose 20%, and pull requests rose 23%. But shipped software output didn't budge. Review process time climbed 49%, the share of pull requests requiring changes nearly doubled, and issue resolution rates showed no statistically significant change.

The researchers' explanation is blunt: the efficiency gains were "absorbed by downstream constraints in the production process." The bottleneck didn't disappear — it moved to code review. Humans suddenly had 30% more code to vet, and the review queue swallowed every minute the agents saved.

This isn't an isolated anomaly. Self-reported numbers tell a similar story of modest real-world impact. AI users report saving an average of 5.4% of their work hours — roughly 2.2 hours per week — and researchers at the St. Louis Fed estimate generative AI contributed a potential 1.1% increase in U.S. productivity by the second half of 2024. Meanwhile, 66% of organizations report productivity gains from AI, but only 20% report increased revenue.

So which is it: 126% gains or barely measurable output? The answer depends entirely on where you place the measuring stick:

• Task-level measurement captures the AI-accelerated moment — writing the document, resolving the ticket, generating the code. This is where controlled studies shine.
• End-to-end measurement captures the whole workflow, including the human steps downstream — review, revision, approval, integration. This is where the Harvard/Jellyfish study found gains evaporating.
• Organizational measurement captures business outcomes like revenue and shipped product — where only a fifth of companies report improvement.

The lesson isn't that AI fails. It's that a 66% gain at one step of a workflow tells you almost nothing about the gain across the whole workflow. MIT Sloan researchers put it well: AI ROI "still often feels more like art than science: elusive, imprecise, and industry-dependent."

That's why measurement method matters as much as the tool itself. Claims about productivity gains are frequently self-reported, vendor-driven, or measured at only the most flattering point in the process. This is the gap SkillzBench is building its benchmarking to address: testing every agent skill in a sandbox with the same model, with and without the skill, and publishing the Lift in percentage points — so before/after comparisons come from controlled tests, not marketing.

Before betting workflow productivity on any AI tool, the evidence says: demand the whole-pipeline number, not the task-level highlight.

Why the Numbers Conflict: Task Speed Isn't End-to-End Productivity

How can AI deliver a 66% speed boost in a lab yet leave company output barely changed? The answer lies in a distinction most ROI estimates miss: the difference between finishing a task faster and shipping the whole workflow faster.

The clearest evidence comes from a Harvard/Jellyfish study of 700+ software firms. AI coding agents produced 30% more lines of code and 23% more pull requests — yet issue and epic resolution rates showed no statistically significant change. As the researchers put it, efficiency gains were absorbed by downstream constraints: average code review time rose 49%, and the share of pull requests requiring changes nearly doubled. The bottleneck simply moved from writing code to checking it.

Measurement method explains much of the remaining conflict. Controlled studies report a 66% average throughput improvement, while workers' self-reports show only 5.4% of work hours saved — about 2.2 hours a week — and macro estimates put generative AI's contribution to U.S. productivity at just 1.1% by late 2024. Self-reports carry risk in both directions: workers may ease up on saved time, while firm-level automation never shows up in surveys. Meanwhile, MIT Sloan notes there is no standard practice for measuring AI ROI — two nearly identical investments can be judged by entirely different definitions of success.

That measurement gap is why before-and-after testing matters more than vendor claims. It's the principle behind SkillzBench's approach to agent skills: benchmark every skill in a sandbox with the same model, with and without it, and publish the resulting Lift so gains are measured, not assumed.

Who benefits most also varies sharply, and that skews averages:

• Bottom-quintile support agents improved throughput 35% — 2.5x the average — while top performers gained only a few percent.
• In an NBER field study, agents with two months of AI-assisted experience performed as well as peers with six or more months without it.
• Time savings concentrate in math, computing, and information services — and shrink toward zero in personal services and hospitality.

The takeaway for ROI estimation: measure the whole workflow, not the AI-assisted step. A tool that doubles task speed delivers nothing if the downstream review, approval, or integration stage absorbs the gain.

How to Actually Measure AI Productivity: A Practical Framework

Most organizations can't answer a simple question: did AI actually make us more productive? Deloitte's 2026 AI report captures the gap perfectly — 66% of organizations report productivity gains, but only 20% report increased revenue. Something between "we feel faster" and "the business is better" is being lost, and it's usually measurement.

The fix starts before deployment. Establish a baseline — cycle time, cost per outcome, quality scores, revenue per workflow — because once AI is live, attribution becomes impossible without a "before" to compare against. MIT Sloan researchers note there is no standard practice for measuring AI ROI, and two companies making nearly identical investments may define success entirely differently.

Second, measure the whole workflow, not just the AI-assisted step. The clearest cautionary tale comes from a Harvard/Jellyfish study of 700+ software firms: AI coding agents drove 30% more lines of code and 23% more pull requests, but review time rose 49% and issue resolution rates showed no statistically significant change. The efficiency gains were absorbed by the next bottleneck. If you only measure the AI step, you'll celebrate a gain that never reaches the bottom line.

Third, track business outcomes instead of inputs. Hours saved is the vanity metric of AI adoption — the St. Louis Fed finds users self-report saving just 5.4% of work hours (~2.2 hours/week), far below the 66% average throughput gain seen in controlled studies. Time saved only becomes value when it's redirected into measurable results. A practical scorecard looks like:

• Cost per outcome — what it costs to resolve a ticket, produce a document, or ship a feature, before and after AI.
• Quality — error rates, rework, and review burden, not just volume.
• Revenue or output — did throughput actually increase end-to-end, or did the bottleneck just move downstream?
• Adoption and time-to-value — how quickly the tool reaches production and real use.

Finally, demand with-and-without comparisons rather than vendor claims. The gap between controlled results and self-reports shows that measurement method matters enormously — which is exactly why SkillzBench is building independent benchmarking that tests every agent skill in a sandbox with the same model, with and without the skill, and publishes the Lift in percentage points. When you evaluate any AI tool or skill, insist on before/after evidence under identical conditions. If a vendor can only offer testimonials, treat the productivity claim as unproven.

Measure this way and the Deloitte gap starts to close: you'll know not just whether people feel faster, but whether the business actually is.

From Measurement to Deployment: Picking AI Tools and Agent Skills That Deliver

A team can buy every AI tool on the market and still see no productivity lift. Gallup's Q4 2025 workplace research found that the top barrier to individual AI use is not tool availability but lack of utility — workers simply don't see a job the tool can do for them.

That insight reshapes deployment. Start with the task, not the tool.

Target explicit use cases, especially where novices struggle. The strongest controlled results come when AI compresses experience gaps. In the NBER field study, agents with two months of AI assistance performed at the level of unassisted peers with six or more months on the job. Bottom-quintile support agents in the NN/g studies improved throughput 35% — roughly 2.5x the average gain. Pick workflows where a clear, repeatable prompt can lift a less-experienced employee toward expert output: first-pass customer replies, boilerplate-heavy business writing, routine code generation, data cleanup.

Measure the whole workflow, not the AI step. The Harvard/Jellyfish study of 700+ software firms showed AI coding agents drove +30% lines of code, +20% commits, and +23% pull requests — but review time rose 49% and issue resolution rates showed no statistically significant change. The bottleneck moved downstream. If you only clock the AI-assisted task, you will overstate the win.

Vet what you install. Agent skills — the instruction packages that customize Claude Code, Codex, Cursor, and other coding agents — are proliferating on GitHub, and published research has found roughly 26% of them carry at least one vulnerability. Quality and security vary widely across the ecosystem, so version pinning, license checks, and sandbox testing belong in any deployment checklist.

That is the gap an independent benchmark directory can close. SkillzBench is building a free platform that crawls and deduplicates agent skills, pins every version to a commit SHA, security-scans and quality-judges each one, then runs each skill in a sandbox with the same model — with and without the skill — to publish the productivity Lift in percentage points. Trust scores (0–100, covering security, quality, license, maintenance, and community signals) stay separate from Lift, and a free MCP connector (in development, waitlist open) lets any agent load only the skills you choose.

For readers ready to move from measurement to deployment, the practical sequence is short:

• Pick one specific task where novices lose the most time
• Baseline current throughput and quality before turning the tool on
• Measure downstream steps too — review, rework, handoff
• Demand before/after benchmarks for any skill you install, not vendor claims

Skills, proven. Join the SkillzBench waitlist to benchmark and compare agent skills before betting workflow productivity on them.