Metrics 3.0 - Meaningful Measurements for Agile Software Development

Metrics 3.0
2017 Mile High Agile - Denver
Presented by Andy Cleff
Co-Author Ralph van Rosmalen
Based on work of Jurgen Appelo

Metrics 3.0 • Andy Cleff • @JustSitThere
Overview
Who’s Here Today?
Twelve Rules  
for Measurement
Group Brainstorming
Group Presentations
Keeping in Touch

12 Rules for Measurement
1: Measure for a purpose
2: Shrink the unknown
3. Seek to improve
4: Delight all stakeholders
5: Distrust all numbers
6: Set imprecise targets
7: Own your metrics
8: Don’t connect metrics to rewards
9: Promote values and transparency
10: Visualize and humanize
11: Measure early and often
12: Try something else
change and innovation practices
MANAGEMENT 3.0
When selecting metrics, ask:
Rule 1: Measure for a purpose
You must always understand why you are measuring.
The metric is not a goal in itself. Never forget that it’s
just a means to an end. It all starts with why.
Rule 2: Shrink the unknown
A metric is just a surrogate for what you really want to
know. Don’t jump to conclusions. Always try to reduce
the size of what is still unknown.
Rule 3. Seek to improve
Don’t only measure things that will make you look good.
There is plenty of data around, but you must focus on
what enables you to do better work.
Rule 4: Delight all stakeholders
Your work depends on others, and others depend on
you. Never optimize for just one stakeholder. Instead,
measure your work from multiple perspectives.
Rule 5: Distrust all numbers
Observers usually influence their own metrics, and they
suffer from all kinds of biases. Have a healthy, skeptical
attitude towards any reported numbers.
Rule 6: Set imprecise targets
When people have targets, they have an inclination to
focus on the targets instead of the real purpose. Avoid
this tendency by keeping your targets vague.
Rule 7: Own your metrics
Everyone is responsible for their own work, and metrics
help us improve that work. Therefore, everyone should
be responsible for their own metrics.
Rule 8: Don’t connect metrics to rewards
Rewards often kill intrinsic motivation and lead
to dysfunctional behaviors in organizations. Don’t
incentivize people to do work they should like doing.
Rule 9: Promote values and transparency
Human beings are smart and able to game any system.
To prevent gaming, be transparent about values,
intentions, and the metrics everyone is using.
Rule 10: Visualize and humanize
Numbers tend to dehumanize everything. Replace digits
with colors and pictures, and keep the measurements
close to where the actual work is done.
Rule 11: Measure early and often
Most people don’t measure often enough. Measure
sooner and faster to prevent risks and problems from
growing too big for you to handle.
Rule 12: Try something else
It’s rarely a good idea to do the same things over and
over. The environment changes all the time. The same
should apply to how and what you measure.
› Why “this metric?” – Why does it matter?
› What insights might we gain from it?
› What is expected to change? What is expected variability,
consistency – are we looking for trends or absolute values?
› How might it be gamed, misused (or abused)?
› What are some for trade offs / costs of improvement -
Working to improve one thing may temporarily reduce
another (e.g., predictability may increase at the expense
of throughput)
› How often would we like to “take a data point”?
› How long will we run the experiment? (What is the half-life?)
› How when we know when we’re “done” with this metric
(and it’s served its purpose, and it’s time to retire it and
consider another…)?
› How will we make our measurements transparent – to
promote knowledge sharing, collaboration with other
teams and trust with our sponsors?
› Is this metric a leading or lagging indicator?

Rule 1- Measure
for a purpose
You must always understand what
you are measuring. The metric is
not a goal in itself. Never forget that
it’s just a means to an end. It all
starts with why.

“If all we have are opinions, let’s go with mine.”
Jim Barksdale
“…Analysis without numbers is only an opinion.”
Atkins Law #1

Reasons why we do measure
To see revenues to drive resource & people allocation
Monitor alignment with mission / vision / goal
Observe quality of product / process
Judge customer happiness / employee satisfaction
To make decisions that are not based on gut feelings

Reasons why we don’t measure
Measurements might be used as weapons
Lame metrics that would not useful or actionable
Implementing measures would cost too much time / effort
Some things might just not be immeasurable

Rule 2 - Shrink
the unknown
A metric is just a surrogate for what
you really want to know. Don’t jump
to conclusions. Always try to reduce
the size of what is still unknown.

Cynefin Framework

Cynefin Framework
Ordered
Domains

Cynefin Framework
Here Things  
Get Interesting…

Rule 3 - Seek to
improve
Don’t only measure things that will
make you look and feel good. There
is plenty of data around, but you
must focus on what enables you to
do better work.

Actionable Metrics
“A good metric changes the way you behave. This is by far
the most important criterion for a metric: what will you do
differently based on changes in the metric?”
Lean Analytics, Alistair Croll and Benjamin Yoskovitz

Vanity Metrics
“When we rely on vanity metrics, a funny thing happens.
When the numbers go up, I've personally witnessed everyone
in the company naturally attributing that rise to whatever they
were working on at the time. That's not too bad, except for this
correlate: when the numbers go down, we invariably blame
someone else”
Eric Ries

Rule 4 - Delight
all stakeholders
Your work depends on others, and
others depend on you. Never
optimize for just one stakeholder.
Instead, measure your work from
multiple perspectives.

It is impossible to please everyone, but you would like to
know who is pleased at certain moments and who is not.

Rule 5 - Distrust
all numbers
Observers usually influence their
own metrics, and they suffer from
all kinds of biases. Have a healthy,
skeptical attitude towards any
reported numbers.

Story
Hawthorne Works, Chicago, ca. 1925.

Rule 6 - Set
imprecise targets
When people have targets, they
have an inclination to focus on the
targets instead of the real purpose.
Avoid this tendency by keeping
your targets vague.

"When a measure becomes a target, it ceases to be a
good measure."
Goodhart's law

Rule 7 - Own
your metrics
Everyone is responsible for their
own work, and metrics help us
improve that work. Therefore,
everyone should be responsible for
their own metrics.

Important Considerations
How many metrics should a team use?
Which ones to use?
How long should they use the ones selected?

Anti-Patterns
Looking at a single metric (Hawthorn)
Striving for ever increasing values instead of striving for
consistency and stability (Goodhart)
Correlation is not necessarily causation (Milton Friedman’s
Thermostat)
Comparing metrics across teams that are very different

Rule 8 - Don’t
connect metrics
to rewards
Rewards often kill intrinsic
motivation and lead to dysfunctional
behaviors in organizations. Don’t
incentivize people to do work they
should like doing.

Rule 9 - Promote
values and
transparency
Human beings are smart and able
to game any system. To prevent
gaming, be transparent about
values, intentions, and the metrics
everyone is using.

Values, Intention, Purpose
Do we get paid a sustainable value for what we do?
Are we great at what we do in the eyes of our customers?
Do our employees / team mates love what we do and the
way we do it?
Will what we do make the world a better place for our
grandchildren?

Rule 10 -
Visualize and
humanize
Numbers tend to dehumanize
everything. Replace digits with
colors and pictures, and keep the
measurements close to where the
actual work is done.

Rule 11 -
Measure early
and often
Most people don’t measure often
enough. Measure sooner and faster
to prevent risks and problems from
growing too big for you to handle.

“The only way to win is to learn faster than anyone
else”
Eric Ries
“What you want to do as a company is maximize the
number of experiments you can do per unit of time.”
Jeff Bezos

Rule 12 - Try
something else
It’s rarely a good idea to do the
same things over and over. The
environment changes all the time.
The same should apply to how and
what you measure.

Limited Lifespan of all Metrics
That which is measured will improve, at a cost.
When a measure becomes a target, it ceases to be a good
measure.
Correlation is not causation, but it sure is a hint.
Use multiple viewpoints - technical as well as human - to get
a holistic perspective

The Twelve Rules for Metrics
1: Measure for a purpose
2: Shrink the unknown
3. Seek to improve
4: Delight all stakeholders
5: Distrust all numbers
6: Set imprecise targets
7: Own your metrics
8: Don’t connect metrics to rewards
9: Promote values and transparency
10: Visualize and humanize
11: Measure early and often
12: Try something else

Five Categories of Metrics
1. Process Health Metrics - assess day-to-day delivery team activities and
evaluates process changes.
2. Release Metrics - focus on identifying impediments to continuous delivery.
3. Product Development Metrics - help measure alignment of product
features to user needs.
4. Technical / Code Metrics - help determine quality of implementation and
architecture.
5. People/Team - reveal issues that impact a team’s sustainable pace and
level of engagement.
Shout out to Jason Tice @theagilefactor

Review the Options….
1. Why “this metric?” – Why does it
matter? Who does it matter to?
2. What insights might we gain from it?
3. What is expected to change? Are we
looking for variability, consistency,
trends or absolute values?
4. How might it be gamed, misused (or
abused)?
5. What are some trade offs / costs of
improvement?
6. How often would we like to “take a
data point”?
7. How long will we run the experiment?
8. How when we know when we’re “done”
with this metric?
9. How will we make our measurements
transparent?
10.Is this metric a leading or lagging
indicator?

Let’s keep the conversation going…
Andy Cleff
andycleff@icloud.com
andycleff.com
linkedin.com/in/andycleff
@JustSitThere
coalition.agileuprising.com

40+ Metrics for Software Teams
People/Team: Human Elements
This group of metrics reveals issues that impact a team’s
sustainable place and level of engagement.
›› Team Happiness / Morale / Mood
›› Gallop Q12
›› Team / Manager / Organization NPS
›› Percentage of time w/o interruptions
›› Trust between Leadership and Team
›› Learning Log
›› Team Tenure
›› Phone-a-Friend Stats
›› Whole Team Contribution
›› Transparency (access to data, access to customers,
sharing of learning, successes and failures)
›› Comparative Agility: Team mapping against the 12 agile
principles (Geoff Watt’s “Scrum Mastery”)
Process Health Metrics
This category assess day-to-day delivery team activities
and evaluates process changes.
›› Cumulative Flow Diagrams
›› Control Charts
›› Cycle Time
›› Percent Complete and Accurate
›› Time Blocked per Work Item
›› Story/Epic Lead Time
›› Successful Iteration Completion
›› Escaped Defect Resolution Time
Release Metrics
This group directs focus on identifying impediments to
continuous delivery.
›› Escaped Defects
›› Release Success Rate
›› Release Time
›› Time Since Last Release
›› Cost Per Release
›› Release Net Promoter Score
›› Release Adoption / Install Rate
Product Development Metrics
These help measure alignment of product features to user
needs.
›› Customer / Business Value Delivered
›› Risk Burndown
›› Value Stream Mapping
›› Sales Velocity
›› Product Forecast
›› Product Net Promoter Score (NPS)
›› User Analytics
Technical/Code Metrics
The following help determine quality of implementation
and architecture.
›› Test Coverage
›› Unit/Regression Test Coverage
›› Build Time
›› Defect Density
›› Code Churn
›› Code Ownership
›› Code Complexity
›› Coding Standards Adherence
›› Crash Rate
›› Build Breaks
›› Technical Debt
›› Ratio of Fixing Work vs Feature Work
Andy Cleff
Andy is an experienced and pragmatic agile practitioner that
takes teams beyond getting agile to embracing agile. His chief
weapons are well asked questions, insightful retrospectives and
an ability to withstand awkward silences. And if all else fails, beer.
andycleff@icloud.com
andycleff.com
linkedin.com/in/andycleff
@JustSitThere
agileuprising.com
The following listing is intended as a starting point for conversation and discussion. Choose one or two that make sense for
your team / organization and add them to your current dashboard. Then rinse and repeat over time.

change and innovation practices
MANAGEMENT 3.0
When selecting metrics, ask:
Rule 1: Measure for a purpose
You must always understand why you are measuring.
The metric is not a goal in itself. Never forget that it’s just a
means to an end. It all starts with why.
Rule 2: Shrink the unknown
A metric is just a surrogate for what you really want to
know. Don’t jump to conclusions. Always try to reduce the
size of what is still unknown.
Rule 3. Seek to improve
Don’t only measure things that will make you look good.
There is plenty of data around, but you must focus on what
enables you to do better work.
Rule 4: Delight all stakeholders
Your work depends on others, and others depend on you.
Never optimize for just one stakeholder. Instead, measure
your work from multiple perspectives.
Rule 5: Distrust all numbers
Observers usually influence their own metrics, and they
suffer from all kinds of biases. Have a healthy, skeptical
attitude towards any reported numbers.
Rule 6: Set imprecise targets
When people have targets, they have an inclination to
focus on the targets instead of the real purpose. Avoid this
tendency by keeping your targets vague.
Rule 7: Own your metrics
Everyone is responsible for their own work, and metrics
help us improve that work. Therefore, everyone should be
responsible for their own metrics.
Rule 8: Don’t connect metrics to rewards
Rewards often kill intrinsic motivation and lead to
dysfunctional behaviors in organizations. Don’t incentivize
people to do work they should like doing.
Rule 9: Promote values and transparency
Human beings are smart and able to game any system. To
prevent gaming, be transparent about values, intentions,
and the metrics everyone is using.
Rule 10: Visualize and humanize
Numbers tend to dehumanize everything. Replace digits
with colors and pictures, and keep the measurements close
to where the actual work is done.
Rule 11: Measure early and often
Most people don’t measure often enough. Measure sooner
and faster to prevent risks and problems from growing too
big for you to handle.
Rule 12: Try something else
It’s rarely a good idea to do the same things over and over.
The environment changes all the time. The same should
apply to how and what you measure.
›› Why “this metric?” – Why does it matter? Who does it
matter to?
›› What insights might we gain from it?
›› What is expected to change? What is the expected variability
and consistency – are we looking for trends or absolute
values?
›› How might it be gamed, misused (or abused)?
›› What are some for trade offs / costs of improvement?
›› How often would we like to “take a data point”?
›› How long will we run the experiment? (What is the half-life?)
›› How when we know when we’re “done” with this metric?
›› Are we adding to the dashboard or replacin/retiring
something else?
›› How will we make our measurements transparent – to
promote knowledge sharing, collaboration with other
teams and trust with our sponsors?
›› Is this metric a leading or lagging indicator?

Metrics 3.0 • Andy Cleff • @JustSitThere • AndyCleff.com
Good Ideas / Actionable Metrics
›› Team / Manager / Organization NPS
›› Gallop Q12
Bad / Nasty / Vanity Metrics
›› Time to hire
›› Lines of code per individual
People/Team: Human Elements
This group of metrics reveals issues that impact a team’s sustainable pace and level of engagement.

Process Health Metrics
›› Cumulative Flow Diagrams
›› Lead & Cycle Time
›› Velocity
›› Story points per developer
This category assess day-to-day delivery team activities and evaluates process changes.

Release Metrics
›› Time to first release
›› Frequency of release
›› Lines of code pushed
›› Story points per release
This group directs focus on identifying impediments to continuous delivery.

Product Development Metrics
›› Product Net Promoter Score (NPS)
›› Risk Burndown
›› Number of new features
›› Customer satisfaction survey conducted by sales agent
These help measure alignment of product features to user needs.

Technical/Code Metrics
›› Ratio of Fixing Work vs Feature Work
›› Test coverage
›› Lines of code
›› Causal Analysis
The following help determine quality of implementation and architecture.

Metrics 3.0 - Meaningful Measurements for Agile Software Development

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Metrics 3.0 - Meaningful Measurements for Agile Software Development

Ähnlich wie Metrics 3.0 - Meaningful Measurements for Agile Software Development (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Metrics 3.0 - Meaningful Measurements for Agile Software Development