Product teams are getting better at listening to users and developing a framework of empathy. But how do you effectively translate empathy gained from user research into actionable results?
In this webinar, Archie Miller, Discovery Coach and Chip Trout, Product Design Journey Lead at CarMax will share their tips for coding notes and turning insights into opportunities, solutions, and experiments.
2. About CarMax
• Original used car industry disruptor
• Nation’s largest retailer of used cars
• 190+ stores in 41 states
• Top 10 used car loan originator
• 25,000+ associates nationwide
• FORTUNE 100 Best Companies to Work For® 14 years in a row
14. Actionable Results from UX Research
Learning Velocity
How much research
was done as a team?
Did you GOOB
as a team?
How many experiments
did you run?
15. Actionable Results from UX Research
“Product design documents are
like vacation photos. They help
the people that were there relive
and recall details, but fail to give
people who weren’t there the
same amount or quality of
information.”
Avoid Vacation Photos
16. Actionable Results from UX Research
• Reduces the need for
documentation
• Avoids the telephone game
• Can lessen team debate
• Creates a shared
understanding
• Allows teams to move faster
Avoid Vacation Photos
25. Actionable Results from UX Research
• Have dedicated roles
• Use a framework for notes
• Don’t capture everything
• Allow time for observers to
ask questions
• Debrief while it’s fresh
Taking Notes
26. Actionable Results from UX Research
Imagine you are .
organizing an event
planning to take a trip
shopping for a car
buying a new couch
applying for a job
27. Actionable Results from UX Research
Tell me about
the last time you .
organized an event
took a trip
purchased a car
bought a new couch
applied for a job
28. Actionable Results from UX Research
• Read up on different
techniques
• “That sounds frustrating.”
• “What I hear you say…”
• “So, I hear that’s a question
in your mind.”
Listening is Hard
42. Actionable Results from UX Research
Opportunity Solution Tree
Sell more cars
I want to feel
confident about
my decisions
I want control
over the process
I need to see a
vehicle history
I need to see the
car to understand
it’s condition
I want help but
only when I ask
for it.
I need to know my
options.
43. Actionable Results from UX Research
Opportunity Solution Tree
I want help but
only when I ask
for it.
Solution A
Experiment
Solution B Solution C
45. Actionable Results from UX Research
• Research as a team
• Code your notes
• Find patterns
• Fall in love with your problem space
• Measure through experiments
In Summary
Here is just a little info about CarMax the company. We are are coming up on our 25th anniversary pretty soon and will be excited to hit such an important milestone as a company. Our Associates play a huge role in our Product org – not only do we design the customer experience you see when visiting our site or our native apps, but we also do a lot of work on our Associate facing systems as well and they are great at allowing us to test new things in store along side of them.
As I mentioned I’ve been here 4 years and we’ve been able to assemble a really talented of Product and Visual System Designers. I think it’s important to spend a few minutes discussing how we are organized so that when we get into what good looks like for team norms you have some idea of what a “team” looks like.
This shot from our office in Downtown Richmond where many of our Product teams are located and we realized that we were continually telling our Product story to new hires, visitors and other teams from within the Company. So we decided to dedicate a wall to telling that story and it explains how we are organized and how we work.
I’m going to start with the Core Team and what that means. A core team, sometime called a durable team, is a group consisting of a Product Manager, Product Designer and a Lead Developer. These Three Amigos are dedicated to a particular part of the customer or associate journey. I’ll talk a little more about that later. These 3 are co-located so that collaboration and rapid iteration can happen more easily.
Then surrounding the core team is the whole team. These individuals may support multiple core teams. Those roles include QA, Delivery Managers, Data Analysts, Visual Designers, etc. But when it comes to conducting Discovery – it should ALWAYS included the core team – and any whole team members that are available.
These core teams sit within one of our Customer Journey Stages. Search / Shop / Buy / Sell. Again they are dedicatde to an opportunity space within a Journey stage but often are required to collaborate and co-design with teams within and across the customer journey. Our VSD team spans across the entire journey.
So now that you know a little bit about how we are organized – how exactly do we work?
We practice what is commonly referred to as “ Dual track Agile” What that means is Discovery is continuous – opportunities, ideas and problems go through the discovery process and make their way into the delivery backlog where the development team can focus on releasing. If you’re doing it right you toss a lot of ideas in the trash, while saving the learning's. What your discovery process looks like will depend on what you are trying to learn and how quickly you can learn it. I’ve been on teams where we go through several rounds of validation in one day and teams it takes longer because of the nature of the problem. That’s your learning velocity, which I’ll talk about next. What DT isn’t – is a product designer going off on their own and handing back leanings to the team. I’ll talk about what good looks like in a second.
While a product manager, designer, and senior engineer may lead and orchestrate discovery, they must involve the whole team in discovery tasks wherever possible. Keep discovery work and progress visible to the whole team.
Keep measuring and learning even after you ship.
How does this fit into velocity? Speed, direction?
Intro to velocity
The UX wisdom of Yogi Berra “You can observe a lot by watching”
If you are part of an Agile team then you are probably used to hearing the word “velocity” quite a bit. Maybe you review your team’s velocity as part of your sprint planning and retros. Your stakeholders might be asking your delivery managers……
A predictable delivery velocity is important to the health of the team – but in addition to asking “How much did we deliver?” you should also be asking yourself as a team….
“How much did we learn?”.
This is a total vanity metric – and I would only recommend it to teams who are new to Discovery or who have maybe fallen out of habit of doing continuous discovery.
You can start to track and ask yourself, and your team….
How much research was done as a core team?
• Discovery is not the job of the Product Designer to go off an do in isolation – it’s a team sport.
• Here we have a Product Designer, Matt and on the right a Lead Developer, Brenda talking to an Associate – I assume the Chris the Product Manager took the pic.
Did you GOOB – Get out of the building
• We use a lot of remote feedback tools but also spend a good amount of time in the field with our customers and associates. Many of the tools we design should be viewed in the context to how they are used. If it’s something our Associates use in-store or out on the lot or in the business office – we need to see it in context so that we can account for all of the potential environmental factors that play a role in usage or overall usability.
• In this photo we have the core team, plus the Field Lead.
How Many Experiments Did You Run?
• This team tracked on their evidence wall, something Archie will talk about later, the number of tests they ran for that Sprint.
Now this is a total vanity metric and really doesn’t mean a thing unless you are getting the proper learnings from it. What it does do is it keeps the customer top of mind for the core team and build good habits that leads to healthy teams.
So I mentioned doing all of this as a team – but I want to talk about why it’s important to do so.
For better or worse, this is the way documents actually work.
If you participate in lots of discussions about what software to build, and then create a document to make sense of it, you might share it with someone else who was there. You might both agree it’s good. But remember, your shared understanding is filling in details that aren’t in the document. Another reader who wasn’t there won’t get the same things from it that you will. Even if they say they get it, don’t believe them. Get together and use the document to tell a story the same way I used my vacation photo to tell you my story.
So at a minimum the core team should all participates in Discovery – when that happens you…
Method
Testing for Value - Would you use it versus could you use it?
Testing for value vs testing for usability
Method
Testing for Value - Would you use it versus could you use it?
Testing for value vs testing for usability
Method
Harvesting insights
Insights
This is really important. You don’t need to capture everything. You only need to capture the information relative to your learning agenda. If you record the session, you will never listen to that recording.
Facilitator should avoid taking notes. At least one person for note taking but hopefully the entire team can contribute.
Difference between what you want to learn and what you need to ask.
Instead of using the commonly used tactic of “Imagine you are (insert action). As soon as you say “imagine” you are giving that person liberty to make something up and you are running the risk of not getting an authentic response. This approach might be fine in the “could they use it” category of usability testing when the UI is more what you are seeking feedback on. However when doing early discovery – try instead to get your participants to tell you a story,
By having customers tell you a story you get to hear the process in their own words and in their own steps. What you assume or think they do is often very different than what they actually do.
Insights
Listening - active listening, mirroring, summarizing
"That's sounds frustrating"
"What I heard you say …"
If they ask you "is this right?"
"so, I hear that's a question in your mind."
Insights
Yummy Sounds - behavioral clues, non-verbal expressions of delight
Insights
Coding Notes - using a matrix for categorizing notes
Insights
Coding Notes - using a matrix for categorizing notes
Insights
Syntheses between each session, don’t wait until the end.
So you’ve collected insights. And you have a pile of stickie notes now what?
Understanding
Synthesizing insights - finding patterns across sessions, Delta-next method
Understanding
Finding pattern from listening sessions and synthesizing into a delta next
Understanding
Delta Next example
Understanding
Finding pattern from listening sessions and synthesizing into a persona
Understanding
You can build your persona within framework that fits your problem space. In this example, the team wanted to align a persona with known data from a broader segmentation study. They also wanted to look at the problem space through the lens of Jobs To Be Done – why would Andy hire CarMax?
You can use any framework, the important thing is that you only include insights that are true assumptions and are ”differences that make a difference"
Added animation to highlight top left
Understanding
Usability issues - how to track usability issues while testing for value.
Results
Mapping Problem Space - fall in love with the problem, not the solution, sizing and prioritizing Opportunities
Results
Size and proritize
Conversations with stakeholders
Results
Crime scene boards no two would be the same