Ask an engineering team whether AI tools have made them faster, and most will say yes. Many will say yes with real enthusiasm.
Then ask a second question: how much faster, compared to what?
The room usually goes quiet.
Most companies adopted AI tools quickly, and for good reasons. The tools are useful. But very few measured where they stood before. So now, when leadership asks whether the investment is working, the honest answer is often "we think so."
That is not good enough for a decision that affects how the whole team works, how much it spends, and how much risk it carries.
Feeling Faster Is Not the Same as Being Faster
AI tools change the experience of work. Writing code feels quicker. Boilerplate disappears. Answers arrive in seconds. It is natural to feel more productive.
But the feeling and the result can drift apart. Code may be written faster and reviewed more slowly. More changes may be produced, with more of them needing fixes later. Individual tasks may speed up while the overall time from idea to customer stays the same, because the bottleneck was never typing.
None of this means the tools are not helping. It means you cannot know from the feeling alone.
What to Measure
A baseline does not need to be complicated. It needs to be honest, and it needs to be taken before, or as early as possible in, the change.
Lead time How long does it take for a small change to go from started to available to customers? This shows whether faster coding becomes faster delivery.
Change failure rate What share of releases need a fix, a rollback, or an urgent patch? This shows whether speed is coming at the cost of quality.
Review load How long do changes wait for review, and how large are they? AI tools often increase the volume of code. If review becomes the bottleneck, the gains disappear.
Unplanned work How much of the team's time goes to incidents and urgent fixes? If it grows after adoption, something in the process is not keeping up.
Onboarding time How long does it take a new engineer to deliver something meaningful? AI tools can help a lot here, or they can hide gaps in understanding.
You do not need perfect data. A few weeks of consistent measurement before and after is far better than none.
If You Already Adopted Without a Baseline
Most teams are in this position. It is not too late.
- Start measuring now. Today's numbers become your baseline for the next stage. You will not know exactly what the first wave of adoption changed, but you will know what the next changes do.
- Use history where you can. Version control, ticket systems, and release logs often hold enough data to estimate the old lead time and failure rate. It will be rough, but rough is useful.
- Compare like with like. If some teams or projects adopted later, or use the tools differently, compare them carefully. Different work makes comparisons weak, so be honest about the limits.
- Ask specific questions. Instead of "do you like the tools?", ask where they save time, where they create extra work, and where they make mistakes that someone has to catch.
Measure the Guardrails Too
AI adoption is not only about speed. It also changes risk.
Generated code can include security problems, licensing questions, or subtle errors that look correct. Sensitive data can end up in prompts. Judgment can slowly move from the team to the tool without anyone deciding that it should.
So alongside productivity, track the guardrails: how many issues are caught in review, how many reach production, and whether the team's own understanding of the system is growing or shrinking. A team that ships faster but understands less is building a problem for later.
Make the Decision With Evidence
The point of a baseline is not to prove that AI tools are good or bad. It is to make better decisions about them.
With real numbers, you can see where the tools help most and invest there. You can see where they create hidden costs and add guardrails. You can tell leadership, with evidence, what the change achieved, and plan the next step with confidence.
Without them, you are left with enthusiasm on one side, scepticism on the other, and no way to settle the argument.
Measure first. Then let the results, not the feeling, tell you what is working.
