Main takeaways: - Knowing what metrics to measure and how to measure them are key skills for a Product Manager. Interviewers are always going to gauge this aspect. - How should we think about setting Product Metrics for every situation? How should we think about measuring these? - What are the strengths and limitations of A/B testing. When can you use it and when should you rely on other methods? What are the different methods for measuring metrics and when to employ those.

1. www.productschool.com
How to Think Product Analytics in PM
Interviews by Amazon Sr PM

2. Join 35,000+Product
Managers on
Free Resources
Discover great job
opportunities
Job Portal
prdct.school/PSJobPortalprdct.school/events-slack

3. C O U R S E S
Product
Management
Learn the skills you need to land
a Product Manager job

4. C O U R S E S
Coding
for Managers
Build a website and gain the
technical knowledge to lead
software engineers

5. C O U R S E S
Data Analytics
for Managers
Learn the skills to understand web
analytics, SQL and machine learning
concepts

6. C O U R S E S
Learn how to acquire more users
and convert them into clients
Digital Marketing
for Managers

7. C O U R S E S
UX Design
for Managers
Gain a deeper understanding of your
users and deliver an exceptional end-to-
end experience

8. C O U R S E S
For experienced Product Managers
looking to gain strategic skills needed
for top leadership roles
Product
Leadership

9. C O U R S E S
Corporate
Training
Level up your team’s Product
Management skills

10. Vivek Pandey
T O N I G H T ’ S S P E A K E R

11. Thinking Product Analytics
Vivek Pandey

12. 1. Who is this guy?
2. Why should I listen to him?
3. Does he have dog pictures? (Oh yes!)

13. Vivek Pandey

14. What will we talk about today?
Analytics +
Product Mgmt
● Role of analytics in Product Management
● What does the interviewer *really* want to
know?
Analytics in PM
Interviews
● Structure of questions and why they are
asked
● Opportunity sizing
Measurement &
Metrics
● Funnel Design, A/B tests

15. What will we NOT talk about today?
Product
design
● Design an alarm clock for cats
Fermi
problems
● How many golf balls will fit into the Space
Needle?
Product
strategy
● Should Google launch a LinkedIn competitor?

16. Part.01
Analytics as a part of Product Management

17. What does a
Product Manager
do?

18. What does a Product Manager do?
Ideates
Design new products, services, features, and
improvements
Prioritizes Which ideas to build and implement?
Persuades
Convince leadership, engineering,
and other stakeholders
Executes
Build and launch products.Track,
and manage performance

19. What is the interviewer looking for?
Can you identify the levers that can be
controlled?
05
Can you use data to persuade others?03
Can you correctly interpret data to make
decisions?02
Can you identify what data is relevant?01
Are you realistic about what data can be
collected?
04

20. Part.02
Analytics Questions and how to think about them

21. I group Product analyses into four buckets
Success definition and measurement03
Prioritization02
Opportunity sizing01
Diagnosing issues04

22. Example Questions
Bucket Typical Question Structure
Opportunity sizing ● Should we launch product/ feature X?
● How many Italian restaurants are in Seattle?
Prioritization ● Given X amount of time/money, which ideas should be prioritized?
● Of all the features we discussed, which one would you build?
Success definition and
measurement
● Pick one metric to manage the feature you designed?
● How will you measure success after you launch your idea?
Diagnosing issues ● Ad revenue dropped by 20%; how will you identify the issue?
● Why are ‘Product Page Views’ down?

23. Product analysis: Opportunity sizing
Identify purchase price05
Identify addressable segments03
Identify the base population02
Clarify scope01
Identify purchase frequency04

24. Let’s run through an example:
Should we launch a bicycle-based
food delivery service in Seattle?

25. There are two parts to the answer
Also to consider (not covered):
● Will this be profitable?
○ Competitors
○ Achievable market share
○ Unit economics
○ Does this align with the company’s strategy?
○ Will you find enough riders/delivery-people?
What is the size of the opportunity?

26. Going through the product analysis steps:
In this case, we are only talking about Seattle.
Clarify sources of revenue. Delivery fee? Commissions?
Clarify scope01

27. Who has access to your service? Base your calculations on a geographical
area, industry, or demographic.
What’s your unit of measure for your customers? Individuals? Households?
Which measure makes most sense for a service like this?
Identify the base population02

28. What’s your unit of measure for your customers? Individuals? Households?
Which measure makes most sense for a service like this?
Identify the base population02
Answer:
# of Households = Population of Seattle / Average Household Size
so.. 800K Individuals / 2.5 Individuals per household = 320K Households

29. This is the part where you show your creativity!
● How are you identifying your segments? By geography? By income?
○ How big is the serviceable area? What are the constraints?
○ How many households are in serviceable areas?
○ How many households order in? How many can afford to?
● So… what fraction of the base population is addressable?
Identify addressable segments03

30. So… what fraction of the base population is addressable?
Answer:
Bicycle speed is a limitation, so we are starting with tight population and
restaurant clusters. Let’s assume serviceable areas are just Downtown,
Ballard, Green Lake, and Capitol Hill.
I will assume top 50% households by income are ordering in.
Adjusting for all these factors, about 30K households are serviceable.
Identify addressable segments03

31. How many times will people use your service?
● Pick a reasonable unit of measure: Weekly? Monthly?
○ For some services it gets very intuitive.
● What assumptions are you making?
○ What are you basing them on?
● Will different segments have a different purchase frequency?
○ Would singles order in more often that families with kids?
Note: For physical products, are you considering replacement sales?
Identify purchase frequency04

32. How many times will people use your service?
Answer:
Let’s assume people who order in are doing so 3 times a month on
average. So, now the service will be used 30K times 3 = 90K times per
month.
Identify purchase frequency04

33. What is the unit price?
● Pick a reasonable price.
○ Base it on industry standards, if similar products exits.
○ Base it on the next best option, it it is something new to world.
Identify purchase price05

34. What is the unit price?
Answer:
$5 Per delivery sounds reasonable? (Based on UberEats, Amazon
Restaurants etc.)
Identify purchase price05

35. Total Households = 320K
Addressable Households = 30K
Monthly Purchase Frequency = 3
Revenue Per Delivery = $5
Total Monthly Revenue = 30K * 3 * 5 = $450K Monthly
Or $450K *12 = $5.4M annually
Bringing it together:

36. $5.4M per year
● Competitors (Uber Eats, Amazon Restaurants, DoorDash etc.)
● What is our competitive advantage?
● What is the achievable Market Share
● Unit Economics per delivery
● Does this align with the company’s strategy?
● Risks?
So, should we do it? ¯_(ツ)_/¯
What is the size of the opportunity?
Will this be profitable?

37. Summary: Opportunity sizing
Identify purchase price05
Identify addressable segments03
Identify the base population02
Clarify scope01
Identify purchase frequency04
Of course, discuss competition, seasonality, regional difference, profitable vs.
unprofitable segments etc. to complete your answer.

38. Prioritization, Success Metrics, and Issue
Diagnosis
Questions around these topics -
● Of all the ideas we discussed, which one would you implement first?
● If you had to pick one metric to manage this feature, which one would it
be?
● How will you measure performance of the feature you described?
● Revenue/visits/deliveries dropped by 20%; how will you identify the
issue?
All these questions have one common theme.

39. How well do you understand ‘the funnel’?
Total Site
Visitors
100%
View Item
Detail Page
70%
Hit ‘Add to
Cart’ button
20%
Go to
‘Checkout’
7%
Complete
Purchase
3%
This example depicts a simple eCommerce website’s funnel

40. Inputs and outputs
Total Site
Visitors
Inputs:
Marketing
SEO
Seasonality
100%
Visit Item
Detail Page
Inputs:
# Items shown
Relevance
Price
UX quality
70%
Hit ‘Buy
Button’
Inputs:
Product
description
# of Images
Product ratings
and reviews
20%
Go to
‘Checkout’
Inputs:
Calls to action
Promotions
7%
Complete
Purchase
Inputs:
Card on file
# of fields to
complete
Registered user
vs guest
3%

41. PMs design features to improve inputs
Total Site
Visitors
Inputs:
Marketing
SEO
Seasonality
100%
Visit Item
Detail Page
Inputs:
# Items shown
Relevance
Price
UX quality
70%
Hit ‘Buy
Button’
Inputs:
Product
description
# of Images
Product ratings
and reviews
20%
Go to
‘Checkout’
Inputs:
Calls to action
Promotions
7%
Complete
Purchase
Inputs:
Card on file
# of fields to
complete
Registered user
vs guest
3%

42. Prioritize for impact
View Item
Detail Page
70%
View Item
Detail Page
80%
Feature 1 Improves
Detail Page Views from
70% to 80%
Complete
Purchase
3%
Complete
Purchase
3.5%
Feature 2 Improves
Purchase Completion
from 3% to 3.5%
R
evenue
+14%
R
evenue
+17%

43. Total Site
Visitors
Breakdown:
50% Mobile
50% Web
100%
Visit Item
Detail Page
Breakdown:
60% Mobile
80% Web
70%
Hit ‘Buy
Button’
Breakdown:
15% Mobile
25% Web
20%
Go to
‘Checkout’
Breakdown:
10% Mobile
4% Web
7%
Complete
Purchase
Breakdown:
4% Mobile
2% Web
3%
Troubleshooting - identify the problematic step and narrow down possible causes.
The funnel varies by segment, source of
traffic etc.

44. Part.03
A/B tests

45. Control Treatment
A Bvs.
• A ‘Control’ group of users is presented with
an experience ‘A’ that is unchanged from
the status quo.
• A ‘Treatment’ group is presented with an
alternate experience ‘B’.
• The behavior of these two groups is
compared over time.
A/B Test Primer
B
Select a test Metric Calculate Sample Size Run Test Analyze Results

46. Common mistakes with A/B tests
Picking the wrong test metric01
Financial metrics such as ‘Revenue per order’, ‘Profit’ etc. have a very high natural
degree of variance. These make lousy candidates for A/B testing.
When variance is high, you need a huge sample size. Tests can run for years!
Speed is key, nobody has time for this.
Tip - Pick low variance metrics such as ‘Orders per user’, ‘Click thru rate’
etc. Click thrus and conversion are more telling anyway.

47. Common mistakes with A/B tests
Misunderstanding ‘Statistical Significance’02
You do not run a test until it shows statistical significance.
You run an A/B test until you hit the required sample size and then you check the
results.
At that point -
1. There is a statistically significant difference between Control and Treatment.
OR
1. There is no statistically significant difference between Control and Treatment.
Tip - Use Evan Miller’s A/B testing blog for calculators.

48. Common mistakes with A/B tests
‘Peeking’ at results to draw ‘early data’03
Do not try to read into results until Sample Size is reached.
We plotted the results of 100 experiments. Even though ~60% of them showed
significance at some point, only ONE had a statistically significant difference after
sample sizes were reached.

49. Common mistakes with A/B tests
Ignoring ‘seasonality’04
On a high traffic website (think Netflix, Google, Amazon), I may reach sample size
in a few hours.
But does that give me the full picture?
Users may behave differently over weekends. Make sure you run the test
long enough to capture nuances.

50. When you can’t use A/B tests….
● Legal issues
● Speed / Urgency
● Content based features
A pre/post analysis is a messy but acceptable last-resort alternative.
i.e. measure performance before and after the change, accounting for other
impacts.

51. “If you can’t measure it,
you can’t manage it”
- Peter Drucker

52. “If you can’t measure it,
you can’t manage it”
- Peter Drucker
- William Edwards Deming

53. “It is wrong to suppose that if
you can’t measure it, you can’t
manage it – a costly myth.”
- William Edwards Deming,
(The New Economics, Page 35)

54. Thank you!!!

55. www.productschool.com
Part-time Product Management, Coding, Data Analytics, Digital
Marketing, UX Design, Product Leadership courses and
Corporate Training