Engineering Metrics: A Compass, Not a Report Card

From the field: a sorted table and Can’s 9 PRs

At the start of May I built a dashboard. I meant well: “visibility”. It had three numbers per person: pull requests opened, lines changed, and story points closed in the sprint. I put it on the screen at every Monday meeting. I did not criticise anyone. I just showed it. I did not need to say anything; a sorted table explains itself.

Six weeks later the numbers looked like this:

Nobody cheated. Everyone did what the dashboard rewarded. Pull requests got split and estimates got bigger. The last row was the quietest and the most expensive: reviews were not counted on the dashboard, so reviews stopped happening. Writing “LGTM” and moving on saved time for opening your own pull requests.

Then that Monday came. Can was at the bottom of the table. For six weeks he had been chasing a bug in the withdrawal flow that processed some withdrawals twice. He read three services, reproduced the bug, and opened the fix as one careful pull request. The table showed this as “9”. After the meeting he came to my desk: “Next time I will split that into five PRs.” He was not joking.

The mistake was mine. I removed the dashboard that week, and at the next meeting I apologised to the team. I did not stop measuring. I changed what and who I was measuring.

Why individual metrics get gamed

There is a name for this: Goodhart’s law. When a measure becomes a target, it stops being a good measure. In software this law works especially fast, because engineers are paid to optimise systems. Put a number in front of them and they will optimise that too.

Story points also rest on a misunderstanding. A story point is a unit of estimation, not a unit of productivity. It exists to talk about capacity in sprint planning. Comparing it between people is like putting two different rulers side by side.

The real damage, though, is outside the numbers. The dashboard made every uncounted task invisible: code review, helping a new joiner, fixing an incident at night, reading for three days to find a hard bug. Most of the work that makes a group of people a team is exactly this.

Report card or compass

The same number can be used in two different ways. The difference is not in the number. It is in how you use it:

DORA: four numbers that measure the team

In July I started again, this time with the four metrics from the DORA research. My reason was simple: all four are team numbers. None of them can be split across individuals. One person does not do a deploy, and one person does not solve an incident.

• Deploy frequency: how often you release to production.
• Lead time for changes: how long it takes for a commit to reach production.
• Change failure rate: what percentage of deploys break something.
• Time to restore: when something breaks, how long it takes to bring the service back.

The first two are speed. The last two are stability. Do not separate them. If you only measure speed, the team learns to be fast and fragile. If you only measure stability, never deploying gives you the best score.

You have to write the definitions down, because team metrics can be gamed too. The easiest way is to not count a broken deploy because “it was not an incident, just a small problem”. These are our definitions:

DEPLOY FREQUENCY : releases to production / week
                   source: deploy log. config changes do not count.

LEAD TIME        : first commit -> running in production (median)
                   source: git + deploy log. backlog waiting not included.

CHANGE FAILURE   : broken deploys / all deploys
                   broken = one of these three happened:
                     1) the release was rolled back
                     2) a hotfix shipped within 24 hours
                     3) an incident was opened
                   there is NO "it was just a small problem" exception.

TIME TO RESTORE  : incident started -> user impact ended
                   source: incident record. we measure when the service
                   came back, not when the root cause was fixed.

A number tells you where to look

The first measurement gave a median lead time of 6.5 days. The first reaction in the meeting was “writing code takes too long”. That was everyone’s first guess. When we split the time into steps, the table said something else:

                    JULY     NOVEMBER
writing code         1.2       1.1
waiting for review   1.1       1.0
merge -> production  4.2       0.7    <-- the real difference
-------------------------------------
total                6.5       2.8

# July    : merged work waited for the Thursday release train.
# November: automatic tests after merge + one-click deploy.
# note: each step is a separate median; the total is approximate.

4.2 of the 6.5 days were finished work waiting for the weekly release train. Nobody wrote code faster; the coding step stayed almost the same. The only change was that merged work no longer waited for Thursday. I needed a number to make that decision, and the per-person dashboard never gave me that number.

This is the real job of a metric. A number does not make the diagnosis. It tells you where to look. The team still makes the diagnosis. In the technical strategy post I said a diagnosis should be written with numbers. These four metrics became the first source for those numbers.

What to track

We measure in four-week windows. July is the first measurement, November is the last four weeks:

One warning: two incidents in four weeks is not statistics. It is an anecdote. The average of a 35-minute incident and a 61-minute incident can double next month with one bad night. So we never decide based on one month. If three windows in a row move in the same direction, we call it a trend.

How we use them

• Once a month, 30 minutes, with the team. The team sees the numbers first. The report to management goes with the team’s explanation. The table never goes alone.
• No comparisons. Not between people, not between teams. The lead time of the payments team and the lead time of the mobile team do not measure the same thing.
• No target numbers, only a direction. We did not say “lead time must drop to 2 days”. We asked “where is lead time waiting?” The day you set a target number, Goodhart comes back.
• Not tied to reviews. None of these four numbers goes into anyone’s performance review. The day one does, the compass turns back into a report card.

So how do I see the individual?

When I removed the dashboard, this was the first question, from the team and from above: “How will we know who is doing what now?” The honest answer: if you need a dashboard to know who does what in a team of seven, the missing thing is not data. It is conversation.

The dashboard did not see Can’s six weeks, but I could have. He talked about the same bug in the daily standup every day. The description in his pull request was two pages long. Two teammates had closed a similar problem in their own services thanks to what he found. None of this was a number. All of it was visible to someone who knew where to look.

The way to see a person is still the same: one-on-ones, feedback from colleagues, and reading the work itself. It is slow and it cannot be counted. But it cannot be gamed either. When I gave the numbers to the team’s system and my attention to the people, both worked better.

What did not work for me

• Making the dashboard anonymous. Before I removed the per-person dashboard, I tried hiding the names. In a team of seven, everyone worked out who was who in two days.
• Turning the four metrics into one score. I built a single “engineering health score”. When it dropped, nobody understood why. One number hid all four questions.
• Starting by buying a tool. In the first month we tried a metrics tool. Its default dashboard showed PR count per person again. For the first three months we took the numbers from the deploy log and the incident records, in a simple spreadsheet. Until the definitions are clear, a tool just shows the wrong number more nicely.

Checklist

Conclusion

Can’s 9 pull requests were the most valuable work of those six weeks. The table put him at the bottom, because the table answered exactly the question I had asked it: “Who opened the most PRs?” The number was not wrong. The question was.

The four DORA metrics are not perfect. But two measure speed, two measure stability, and none can be split down to one person, so they are hard to game. More importantly, they showed us that writing code was not the slow part. Waiting for Thursday was.

The test: which question does your metric make your team ask? If the answer is “who is falling behind”, you have a report card. If it is “where are we waiting”, you have a compass.