"How to measure ROI of AI tools and investments"

"Measure AI ROI by tracking three categories: direct productivity gains (time saved per task, not headcount reduced), quality improvements (fewer defects, faster incident resolution, better customer satisfaction scores), and capability expansion (new things you can do that you couldn't before). Ignore adoption rates, lines of code generated, and vendor-provided ROI calculators. Most AI investments should show measurable impact within 90 days — if they don't, the problem is usually implementation, not the technology."

How to Measure the ROI of AI Investments: What to Track and What to Ignore

A client asked me to evaluate whether their $8K/month investment in AI coding tools was paying off. Their engineering manager’s answer: “The team loves it.” That’s great, but it’s not ROI. It’s a satisfaction survey.

The problem with measuring AI ROI is that most companies either don’t measure it at all (they just assume it’s working because the tools feel productive) or they measure the wrong things (adoption rates, number of AI-generated suggestions accepted, vendor-provided benchmarks). Neither approach tells you whether you’re actually getting value.

The Three Categories That Matter

Category 1: Direct productivity gains. This is what everyone starts with, but most measure it wrong. Don’t measure “hours saved” based on self-reporting — engineers are terrible at estimating how long things would have taken without the tool. Instead, measure observable outcomes: Has the time from ticket assignment to first PR decreased? Has the number of PRs merged per sprint increased? Has the time spent on boilerplate tasks (test writing, documentation, scaffolding) decreased relative to time spent on creative problem-solving?

At one client, we measured the average time from feature ticket creation to deployment before and after AI tool adoption. It dropped from 8.2 days to 5.1 days — a 38% improvement. That’s measurable. That’s defensible. That maps directly to business velocity.

Category 2: Quality improvements. Productivity without quality is just faster technical debt. Track: Has the defect rate per deployment changed? Has the mean time to resolution for production incidents improved? Has test coverage increased? Are code review cycles shorter because AI is catching the mechanical issues before human reviewers spend time on them?

The quality metrics matter because they answer the question “are we shipping faster AND better, or just faster?” If your deployment frequency doubles but your rollback rate also doubles, your AI investment is a net negative.

Category 3: Capability expansion. This is the hardest to measure and often the most valuable. What can you do now that you couldn’t do before? One client used AI tools to add comprehensive API documentation for an inherited codebase — something that had been on the backlog for two years because nobody wanted to spend the time. Another used AI-powered security scanning to audit their entire codebase for OWASP vulnerabilities, a project that would have required hiring a security consultant.

Capability expansion doesn’t show up in traditional ROI calculations, but it represents real business value: reduced risk, new product capabilities, faster onboarding for new engineers.

What to Ignore

Adoption rates. If 100% of your engineers use the AI tool and none of them are measurably more productive, you have 100% adoption of a tool that isn’t working. Adoption is a leading indicator at best and vanity metric at worst.

Lines of code generated. More code is not better code. AI tools can generate enormous amounts of code quickly. That’s only valuable if the code is correct, maintainable, and solves a real problem. I’ve seen AI-assisted projects produce 3x more code than a human-only approach — code that was harder to maintain and had more bugs.

Vendor ROI calculators. Every AI tool vendor has a calculator that shows their tool saves $X per developer per year. These calculators assume best-case adoption, optimal use cases, and typically compare against a baseline that doesn’t reflect how your team actually works. Use them for initial justification if you need to, but never for actual measurement.

Time saved on individual tasks. “I saved 20 minutes writing this function” doesn’t tell you anything useful because you don’t know if the developer then spent 30 minutes debugging the AI-generated code, or if the function introduced a subtle bug that cost 4 hours to fix in production.

The 90-Day Rule

If you’ve implemented an AI tool and can’t point to measurable improvement in at least one of the three categories within 90 days, something is wrong. It’s usually one of three things: the tool isn’t matched to your team’s actual workflow (common with AI coding tools imposed top-down), your team hasn’t been trained on effective usage patterns (they’re using it for everything instead of the high-value use cases), or the tool genuinely doesn’t work well for your technology stack and problem domain.

Don’t throw more money at the problem. Diagnose whether it’s a training issue, a workflow mismatch, or a tool limitation — and adjust accordingly.

The Budget Framework

For a company spending $5K-$15K/month on AI tools across engineering, I recommend quarterly ROI reviews that answer three questions: What measurable productivity gains can we attribute to AI tooling? What quality improvements can we document? What new capabilities did AI tooling enable that we wouldn’t have pursued otherwise?

If the answers to all three questions are “not much,” you have an expensive subscription, not a strategic investment.