The document discusses various pitfalls of data-driven development from a product management perspective, focusing on agile software development. It outlines five main pitfalls: 1) having the wrong mindset for failure when conducting experiments, 2) difficulties perceiving progress, 3) overreliance on collecting more data when facing deadlocks, 4) unexpected downsides of democracy in decision making, and 5) lack of domain experts. It emphasizes embracing failures, setting clear success criteria, allowing for random changes, empowering domain experts, and the product manager making the final call.

1. a few words about
Pitfalls of
Data-Driven Development
Product Management
with focus on

2. Product Management
with focus on
a few words about
Customers
a few words about
Developers
Product Management
with focus on
a few words about
Remote Collaboration
Product Management
with focus on
a few words about
Pitfalls of
Data-Driven Development
Product Management
with focus on

3. Bartek Gatz
xmgatz@gmail.com
bartosz.gatz@dynatrace.com
https://www.linkedin.com/in/bgatz/
senior technical product manager

4. Bartek Gatz
xmgatz@gmail.com
bartosz.gatz@dynatrace.com
https://www.linkedin.com/in/bgatz/
mass-market
senior technical product manager

5. a few words about
Pitfalls of
Data-Driven Development
Product Management
with focus on
of a mass-market product

6. Plan of this talk
Thread #1 Thread #2

7. Plan of this talk
Agile Software
Development
Thread #2

8. Inspiration for this talk
https://blog.intercom.com/the-problem-with-data-driven-decisions/

9. Inspiration for this talk

10. Where is the data?
Customer Interviews
Usage Analytics
Support TeamsSales Teams
Business PartnersGrowth Experiments
Competitive Analysis Surveys
In Product Feedback
Collection
Social Media
Conferences
Performance
Analytics
Industry Trends
Public backlogUser groups

11. Where is the data?
Qualitative
Quantitative

12. Where is the data?
Qualitative
Quantitative
Surveys
Customer Interviews
Usage Analytics
In Product Feedback Collection
Conferences
Business Partners

13. Where is the data?
Qualitative
Quantitative

14. Where is the data?
Qualitative
Quantitative
easier to get, but
less valuable
harder to get, but
more valuable

15. Agile Software
Development
Thread #2

17. Typical Agile development behaves
a lot like a Genetic Algorithm

18. Agile Software
Development
Genetic Algorithms

19. Typical Agile
plan
(design)
develop
(build)
deploy
(expose)
measure
(analyze)
repeat
(adapt)

20. Typical Agile
plan
(design)
develop
(build)
deploy
(expose)
measure
(analyze)
repeat
(adapt)
representation fitness function

21. 2017

22. PROBLEM DESCRIPTION (TEXT)
KPI IMPACT HYPOTHESIS (TEXT)
SOLUTION HYPOTHESIS (TEXT)
SUCCESS CRITERIA (TEXT)
LAY TIME STAMP (DATE)
PRIORITY (2x VALUE)
IMPACT COST

23. Fitness Function in Agile
KPI IMPACT HYPOTHESIS (TEXT)
SUCCESS CRITERIA (TEXT)

24. PROBLEM DESCRIPTION (TEXT)
KPI IMPACT HYPOTHESIS (TEXT)
SOLUTION HYPOTHESIS (TEXT)
SUCCESS CRITERIA (TEXT)
LAY TIME STAMP (DATE)
PRIORITY (2x VALUE)
IMPACT COST

25. PROBLEM DESCRIPTION (TEXT)
SOLUTION HYPOTHESIS (TEXT)
Genetic Representation in Agile

26. Evolution Algorithms
representation
target
arena
fitness function = distance from the target

27. Evolution Algorithms

28. Evolution Algorithms

29. Evolution Algorithms

30. Evolution Algorithms

31. Evolution Algorithms

32. Evolution Algorithms

33. Evolution Algorithms

34. Pitfall #1
Mindset problem

35. Mindset problem
Do you have a right mindset
for failure?

36. Mindset problem

37. Mindset problem
90%A/B test fail rate

38. Mindset problem
90% 70%A/B test fail rate A/B test fail rate

39. Pitfall #1
Mindset problem
Embrace the losses

40. Pitfall #2
Perception of progress

41. Are you sure you are going
in the right direction?
Perception of progress

42. Are you sure you are going
in the right direction?
Perception of progress
KPI IMPACT HYPOTHESIS (TEXT)
SUCCESS CRITERIA (TEXT)

43. Evolution Algorithms
obstacle

44. Evolution Algorithms

45. Evolution Algorithms

46. Evolution Algorithms

47. Pitfall #2
hmmm…..
Perception of progress

48. hmmm…..
- „Is there really no progress?”
- „Or we do not see the progress?”
A
B

49. hmmm…..
- „Is there really no progress?”
- „Or we do not see the progress?”
we’ll get back to it
A
B

51. Snorkels, when used by
vehicles with air-
breathing internal
combustion engines,
sometimes allow
limited deep
fording capability for river
crossing or amphibious
landing operations,
particularly in the case of
tanks and other armored
vehicles.

52. How to tell if snorkel is a
needed feature in your
customers’ car?

54. Pitfall #2B
We do not see the progress
Set your success criteria right

55. Pitfall #2B
We do not see the progress
Set your success criteria right
ahead of time

56. Set your success criteria right
ahead of time
KPI IMPACT HYPOTHESIS (TEXT)
SUCCESS CRITERIA (TEXT)

57. Pitfall #3
Reaction to deadlock

58. What happens when you hit a deadlock
in DDD-powered Agile project?
Reaction to deadlock

59. - „Let’s collect more data.”
- „Let’s make safe decisions.”
- „Let’s work faster.”
What happens when you hit a deadlock
in DDD-powered Agile project?

60. What happens when you hit a dead
end in DDD-powered Agile project?
more, safer, faster data
Qualitative
Quantitative
- „Let’s collect more data.”
- „Let’s make safe decisions.”

61. What happens when you hit a dead
end in DDD-powered Agile project?
more, safer, faster data
Qualitative
Quantitative
- „Let’s collect more data.”
- „Let’s make safe decisions.”
unreliable

62. How Genetic Algorithms
escape deadlocks?

63. How Genetic Algorithms
escape deadlocks?

64. Evolution Algorithms

65. Evolution Algorithms

66. Evolution Algorithms

67. Evolution Algorithms

68. What are mutations?
- Seemingly random acts of change
- Dramatic changes as compared
to the rest of the genetic representation
- Chaotic, and potentially unsafe

69. What are mutations?
- Seemingly random acts of change
- Dramatic changes as compared
to the rest of the genetic representation
- Chaotic, and potentially unsafe

70. - Seemingly random acts of change
- Dramatic changes as compared
to the rest of the genetic representation
- Chaotic, and potentially unsafe
- „Let’s collect more data.”
- „Let’s make safe decisions.”
- „Let’s work faster.”
CONFLICT
INTERESTS
OF

71. Pitfall #3
Reaction to
deadlock
There is
no progress
Pitfall #2A
?

72. A role of Domain Expert

73. A role of Domain Expert
- Can extrapolate
from past onto the future
- Someone with the thorough
knowledge of the domain
- Has seen the past and understands
the dynamics of changes

74. What happens when you hit a dead
end in DDD-powered Agile project?
better data
Qualitative
Quantitative

75. What happens when you hit a dead
end in DDD-powered Agile project?
better data
Qualitative
Quantitative
actionable

76. Is Domain Expert a Visionary?

77. Is Domain Expert a Visionary?

78. Is Domain Expert a Visionary?

79. Bernd Greifeneder

80. Evolution of APM
let’s hope
for the best
let’s collect
lots of data
and hope
for the best
let’s collect
lots of data,
improve search,
and hope
for the best
let’s collect
lots of data,
improve search,
add custom reports
and hope
for the best

81. Evolution of APM
let’s hope
for the best
let’s collect
lots of data
and hope
for the best
let’s collect
lots of data,
improve search,
and hope
for the best
let’s collect
lots of data,
improve search,
add custom reports
and hope
for the best
let’s provide
answers
based on
the data
we collect
let’s provide
solutions
based on
the answers
we have in
the data
we collect

82. Pitfall #3
Reaction to
deadlock
There is
no progress
Pitfall #2A
Grow Domain Experts

83. Domain Experts are not always vocal
„We have no domain experts”
„Let’s do a workshop, where everyone
can contribute with their ideas”

84. „Let’s do a workshop, where everyone
can contribute with their ideas”
„Let’s vote on what we will do next”

85. Pitfall #4
Unexpected pitfall
of democracy

86. People are equally right
to share their opinions
People are NOT equally right to make
important decisions
Unexpected pitfall
of democracy

88. Pitfall #4
Unexpected pitfall
of democracy
Product Manager makes the final call
with help of Domain Experts

89. Domain Experts are not always vocal
„We have no domain experts”

90. Pitfall #5
No Domain Experts

91. separate noise from information
„outspoken” does not equal „knowledgeable”
as a PM, listen to your peers.
No Domain Experts

92. Your best Domain Expert friend might be
someone you least expect it to be.
check your customer support specialists
check your oldest and grumpiest developers
check the guys who never wear ties

93. We have no Domain Experts
Can a Visionary prove
his/her vision?

94. Can a Visionary prove
his/her vision?
- It is really, really hard.
- Especially if you only
have cheap data.
Qualitative
Quantitative
- Trust your Domain Experts

95. Evolution Algorithms

96. Evolution Algorithms

97. Pitfall #5
Domain Experts
identification
Let them grow
Let them speak up

98. How to never grow
Domain Experts in your teams?

99. How to never grow
Domain Experts in your teams?
compulsory role
and project rotation,
to increase „bus factor”

100. How to never grow
Domain Experts in your teams?
forceful campaigns
against introverts

101. How to never grow
Domain Experts in your teams?
mindset incompatible
with failure

102. How to never grow
Domain Experts in your teams?
blind belief that
data-driven-development
is the only way

103. How to never grow
Domain Experts in your teams?
„prove it to me”
mentality of managers

105. Agile Software
Development
Genetic Algorithms

106. Seven Year Prediction

107. plan
(design)
develop
(build)
deploy
(expose)
measure
(analyze)
repeat
(adapt)
representation fitness function

108. representation fitness function
If this is an algorithm,
why not automate it?

109. representation fitness function
If this is an algorithm,
why not automate it?
Natural Language Understanding
Automatic verbose feedback collection
Prioritised backlog of experiments
A/B tests for validation

110. representation fitness function
If this is an algorithm,
why not automate it?
Natural Language Understanding
Automatic verbose feedback collection
Prioritised backlog of experiments
A/B tests for validation
80%of development effort by 2025

111. representation fitness function
If this is an algorithm,
why not automate it?
Natural Language Understanding
Automatic verbose feedback collection
Prioritised backlog of experiments
Self-organising software
A/B tests for validation
80%of cost by ??????

112. Thank you!

113. ?