Carol Smith presented "Faster Usability Testing in an Agile World" via webinar during the Agile UX Virtual Summit 2017 by UXPin.
This presentation covers:
- Brief intro to how the IBM Watson Design team runs continuous usability tests and integrates the UX team
- How design work fits into the Agile process via dual track development
- When to run moderated, un-moderated, remote, and in-person studies
- How to effectively communicate UX findings and recommendations
Faster Usability Testing in an Agile World - Agile UX Virtual Summit 2017 by UXPin
1. UXPin Agile UX
Virtual Summit
June 15, 2017
Carol Smith
@carologic
Faster Usability Testing
in an Agile World
2. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Design at IBM Watson
Design Thinking & Agile at IBM with Vera Rhoads
16th June 2017, 2:00-3:00 PM PDT
3. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Researcher and interaction designer on each squad
• Visual designer often shared
• Designers Consult on other teams as needed
Watson - Squads include UX Designers
4. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Iterative
• Repeatable
• Incremental
• Get to 80% confidence
Light Design - Lean UX
5. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Just enough
• Identify and solve problems
• Use each study to pick up information
• Think ahead (not make ahead)
– Concept Cars (vision)
– Know the North Star
No BDUF* at Watson
*Big Design Up Front
6. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Fit testing into Agile process
• Never too early - whatever is ready
• Regular testing
Faster Usability Testing
Image of sign, created by IBM’er Thomas J. Watson
http://www.newyorker.com/cartoons/bob-mankoff/computer-games
7. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Representative users
• Doing real tasks
• Being observed
• Using prototypes
or live products
Usability Testing is…
http://creativecommons.org/licenses/by-sa/2.0/
http://www.flickr.com/photos/raphaelquinet/513351385/sizes/l/in/photostream/
http://www.flickr.com/photos/raphaelquinet/
8. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• There is a difference
• User testing infers testing people
– Can the participant figure out our design work?
• Usability testing puts emphasis on what is being tested
– Reduces the focus on the participant’s performance
– Is the product successful?
Usability NOT User Testing
9. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• All interfaces have usability problems
• Limited resources to fix them
• More problems than resources
• Less serious problems distract
• Intense focus on fixing most serious problems first
True Statements
Adapted from: Rocket Surgery Made Easy:
The Do-It-Yourself Guide to Finding and Fixing Usability
Problems, by Steve Krug
10. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Parallel-Track
Workflows
(How to Think Ahead)
a.k.a. Staggered Sprints
11. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Developer Track:
– Focus is on production
– Whole team (including UX)
• UX Track (Interaction Designer Track)
– Focus is on user contact
Agile Design Timing: Parallel Tracks
“Adapting Usability Investigations for Agile User-Centered Design” by Desirée Sy.
Journal of Usability Studies, Volume 2, Issue 3 http://www.uxpajournal.org/
UX
12. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Iteration 1
“Adapting Usability Investigations for Agile User-Centered Design” by Desirée Sy.
Journal of Usability Studies, Volume 2, Issue 3 http://www.uxpajournal.org/
UX
13. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Iteration 2
“Adapting Usability Investigations for Agile User-Centered Design” by Desirée Sy.
Journal of Usability Studies, Volume 2, Issue 3 http://www.uxpajournal.org/
14. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
And so on…
“Adapting Usability Investigations for Agile User-Centered Design” by Desirée Sy.
Journal of Usability Studies, Volume 2, Issue 3 http://www.uxpajournal.org/
15. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
1) Development work
– Maximize velocity to make working software
– Goal: Working software
2) Discovery work
– Learning velocity - fast learning and validation
– Goal: Learning
Reality Check – 2 Types of Work
Dual Track Development is not Duel Track, May 10, 2017 By Jeff Patton
http://jpattonassociates.com/dual-track-development/
16. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Dual Track Development is not Duel Track
Dual Track Development is not Duel Track, May 10, 2017 By Jeff Patton
http://jpattonassociates.com/dual-track-development/
17. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Between the two tracks
• Essential for success
• Not just hand-offs
• All at once – two tracks, not two teams
• All work will not move from discovery to development
Constant Communication
From Tutorial: Making UX Agile and Lean at UXPA 2016
by John Schrag, Carol Smith and Thyra Rauch
http://uxpa2016.org/event/making-ux-agile-and-lean
18. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Pros
– Space is made for discovery work
• Cons
– Can lead to BDUF and Waterfall
– Not a solo sport - need a team
Pros and Cons - Parallel/Staggered work
19. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Balancing Act
• Understand users, context, etc.
• Create personas, mental models, etc.
• Prepare for story mapping
and other sessions thoroughly
Strive for
UX Best
Practices
• Engineers need designs
to develop
• Research cannot continue forever
Meet
Production
Needs
Agile requires leaner methods
Time box and focus on outcomes
Slide from Tutorial: Making UX Agile and Lean at UXPA 2016
By John Schrag, Carol Smith and Thyra Rauch
http://uxpa2016.org/event/making-ux-agile-and-lean
20. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Continuous improvement
• Continue to prioritize Dev work
• Start small with Discovery work in next sprint
• May need a Design Spike
Existing products/teams
21. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
What to Test
22. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Learn from whatever is ready…
• Concepts, rough ideas,
brainstorming
• Work in Progress
• Prototypes - make to learn
– Clickable (code is optional)
• Websites, mobile, products,
services
• Competing designs (A/B
testing)
• Comparative studies
• Conduct user research
23. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Clickable
– Easy and quick to create
– Easy to change
• Real Code
– Risky – more time invested
– Great if it’s the right solution
• Not paper - participants may misunderstand
Clickable or Code?
24. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
When should I test?
• Early! Any Stage of process
• Anywhere (conference room,
remotely)
• Anytime (un-moderated)
• Realistic test environment
Photo by Roebot at http://www.flickr.com/photos/roebot/2964156413/
25. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Testing Methods
26. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• What do we need to know?
• What gets us to 80% confidence/disproves?
• Bare minimum of effort?
• Place higher value on repeatable formative studies
Adapting Usability Testing Methods
27. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Consider budget, resources
• Time
– Recruiting
– Facilitating
– Analyzing
Scope Effort
28. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Small Scope is Better
• 5-7 participants uncover
~80% of major usability problems
• Don’t try to be statistically
significant
• More participants
- diminishing returns
Jakob Nielsen’s Alertbox. Why You Only Need to Test with 5 Users.
March 19, 2000. http://www.useit.com/alertbox/20000319.html
Number of Test Participants
Participants are representative of specific user group and are doing representative tasks
More participants take more time, cost more money - significantly diminishing returns.
29. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Moderated
– In Person
– Remote (Web Meeting)
• Un-moderated (online tools)
Testing Options
30. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• 3 + participants (tie breaker)
• Short sessions
– Moderated: 30 minutes to 1 hour each
– Unmoderated: Under 20 minutes
Make it Quick
31. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Pros
– Get to “why” by asking follow-up questions
– Work with them longer
– Save or end session as appropriate
• Cons
– Harder to recruit
– Takes longer to plan
Moderated
32. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Pros
– Great for quick answers - complements moderated testing
– Easy recruiting
• Cons
– Recruiting very relaxed
– Professional participants
• Participants camera and internet only
Unmoderated
33. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Remote Testing Options
Moderated Un-Moderated
Researcher is present Researcher not needed during study
Same effort as in-person Minimal effort
Limited number of participants
per day (3-6)
High number of participants per day
Immediate feedback Test data compiled usually within
2 days
Get to the “Why?” by asking
questions
Reduced or no ability to ask
questions
34. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• No travel
• Easy to set up
• Conducted from almost anywhere
• Option to see participants actual environment
• Drawback: Not as high quality as in-person
Benefits of Remote
35. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Robust computer
• Screen sharing software
• Fast Internet on both ends
– May require separate phone line if no VOIP
• Participants camera
– People are becoming more comfortable
– I still do not use
Remote – Moderated Requirements
36. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
More on Remote Testing
• Getting the most out of remote
research and testing
– http://interactions.acm.org/archive/view/march-april-
2017/getting-the-most-out-of-remote-research-and-testing
– ACM Interactions Magazine
by Carol Smith
Getting the most out of remote research and testing. By Carol J. Smith. 2017.
Interactions, 24, 2 (February 2017), 82-84. DOI: https://doi.org/10.1145/3038225
http://interactions.acm.org/archive/view/march-april-2017/getting-the-most-out-of-
remote-research-and-testing
37. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• In-person (office or participant location) or remote?
• Prototype/organization limitations (Online? Clickable?)
• Number of participants, observers?
• Logging and video editing (time on task, highlights)?
• Surveys before or after?
• Eye tracking?
Software Tool Considerations
39. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Nice
– Are your users local?
– Opt for large screen TVs/Cameras vs. two way glass
• Recommend laptop/portable lab
– Can be semi-permanent
– Ready when you are (minimal setup)
Formal Lab?
40. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Make it Repeatable
41. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
“Teams should stretch
to get work into that day’s test
and use the cadence to drive
productivity.”
- Jeff Gothelf
http://blog.usabilla.com/5-effective-ways-for-usability-testing-to-play-nice-
with-agile/
42. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Regular and Early Testing
• Reduces risk and chance
of major issues later
• Priority needs and problems
get priority
• Don’t sweat the small stuff
43. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Small focused tests
• Reduced recruitment time
• Once per Sprint
• Same day(s) mid-week (avoid Mondays and Fridays)
Bring it On!
44. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Team becomes:
– accustomed to steady stream of qualitative insight
– insight encourages quick decisions…line up with business
and user goals
Why Regular?
Adapted from Jeff Gothelf - http://blog.usabilla.com/5-
effective-ways-for-usability-testing-to-play-nice-with-agile/
45. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Problems we will find while usability testing
• Research to inform personas
• Design/UX debt
• Advocate to get prioritized (UX rep should be in planning)
Design Stories
46. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Learn from whatever is ready…
• Concepts, rough ideas,
brainstorming
• Work in Progress
• Prototypes - make to learn
– Clickable (code is optional)
• Websites, mobile, products,
services
• Competing designs (A/B
testing)
• Comparative studies
• Conduct user research
47. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Managing Testing
48. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Select what to test and/or additional work
• Create guide
• Determine recruit and automate where possible
Planning is Tight
49. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Screener
– Technology use/experience
– Knowledge of topic
• Guides/Scripts
• Consent Forms
• Data Collection
Create Reusable Templates
50. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Test & Observation Rooms
– Use repeat calendar feature
• Any location will do
– Conference rooms
– Offices
– Quiet corner of cafeteria
Pre-Book Rooms Quarterly
51. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Recruiting
52. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Hire a Recruiter
• Allows you to focus
• Good recruiters:
– find right participants.
– give regular updates.
– take care of directions,
confirmations, incentives, etc.
53. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Email, phone, etc.
• Create a panel (large recruits, less often)
– Customer lists, user groups
– Schedule as needed
– Use cautiously
• Internal resource who REALLY understands who you need
Do it Yourself
54. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Use Site to Recruit
Ethnio: https://ethn.io/
55. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• “We are all only temporarily able-bodied.
Accessibility is good for us all.”
-@mollydotcom at #stirtrek 2011
• Know the law, build for the spirit of accessibility
– U.S. Section 508 - https://www.section508.gov/content/learn
– WCAG 2.0 - https://www.w3.org/WAI/intro/wcag.php
Include people with disabilities
56. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Keep it easy
– In Person: Gift cards from retail stores
– Remote: Online Gift Cards
• More complex
– Product credits
– Swag (don’t bother)
Honorariums
57. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Make team aware
• Invite everyone
– Can watch remotely
– Recurring meeting invites for stakeholders
Usability Testing Day!
58. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Observing Studies
59. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Stakeholders and decision makers
– PO, PM, Dev, etc.
• Greet participants (in person sessions)
• Take notes and share them
– Create a template if helpful
• Help operate software/recording equipment
Team Members Are Active Observers
60. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Pro’s
– Gain empathy for users
– Understand struggles and what works
– Quick decisions can be made
• Cons
– Time is limited
– Not all sessions are great (but most are!)
Observers
61. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Notetaking
• Keep it simple
• Paper or Computer
– Ask participant if bothered
by typing
– Consider if you can pay attention
(turn off all distractions)
Issue P1 P2 P3
Search Used Yes No No
Widget 1 Used N/A Used –
unsure
about
Task 1 Notes 3 – easy 2 – needed
help
3 – easy
Task 2 Notes 2 – needed
help
2 – easy 2 – needed
help
Task 3 Notes 2 – needed
help
3 – easy Ran out of
time
Task 4 Notes 2 – needed
help
3 – easy Ran out of
time
62. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Analysis, Debrief
and Useful & Usable
Recommendations
63. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Occurs without full analysis when stakeholders have watched
tests
• If not: Quick analysis to quick decisions
Debrief with Team
64. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• All decision makers MUST be present
• Stay on Topic
• Be Constructive
• Don’t get distracted by small problems
• Intense focus on fixing most serious
problems first
Guidelines for Debrief
65. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Identify top 5 or 10 most serious issues
– Top 3 from each observer’s list
– Prioritize from lists
– Commit resources for next sprint
– Stop
Goal
Adapted from: Rocket Surgery Made Easy:
The Do-It-Yourself Guide to Finding and Fixing Usability
Problems, by Steve Krug
66. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Excel - Find Patterns in Data
Learn more: I’m Sorry. I Can’t. Don’t Hate Me:
The Post-It Breakup by Kyle Soucy at IA Summit 2017
https://www.slideshare.net/usableinterface/im-sorry-i-
cant-dont-hate-me-the-postit-breakup
67. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Look for patterns
• Read “between the lines”
• Know what you’ve got
– Sort, reorganize, review, repeat
– What refutes your expectations?
– Surprises?
– Outliers?
Transform Data
68. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Don’t be a tree killer
Image of stack of paper:
https://commons.wikimedia.org/wiki/File:Stack_of_Copy_Paper.jpg
X Marks the Spot https://openclipart.org/detail/193050/x-marks-the-spot
69. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Bulleted list
– Who will use information and how?
– Share via common tools (wiki, Box, etc.)
• Include
– Goal of study
– High level findings and recommendations
– Link to guide and analysis (tasks attempted, actual
performance, etc.)
Just enough
70. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Identify stories related to findings
• Add findings/evidence/recommendations in comments
on related stories in repo
• Tangential findings/evidence/recommendations - also note
in comments
Tie to Stories in Repo
71. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Small iterative changes
– Make it better now
– Don’t break something else
• Take something away
– Reduce distractions
– Don’t just add – question it
Tweak, Don’t Redesign
Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding
and Fixing Usability Problems, by Steve Krug
72. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Faster Usability Testing
• Fit testing into Agile process
• Never too early –
whatever is ready
• Regular testing
https://www.flickr.com/photos/fireatwillrva/5823369990
73. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Contact Carol Smith
in/CarolJSmith
@Carologic
slideshare.net/carologic
74. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Resources, References, etc.
75. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Recommended Readings
7
5
76. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
• Hire great people
• Balance skills and experience
• Enable to work to their strengths
• Encourage to grow, attend webinars, conferences, etc.
• Make time for 1:1’s
Towards Building Teams
77. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Encourage Information Radiators
• User information
• Customer journeys
• Photos from research
• Usability metrics
78. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
Agile Frustrations via Spotify
Scaling Agile at Spotify via Slideshare of Vlad Mysla
http://www.slideshare.net/vmysla/scrum-at-spotify?qid=2345c3ad-7e68-4383-9673-
9e715ff47a75&v=default&b=&from_search=14
squad member
79. Adapting Usability Investigations for Agile User-Centered Design
by Desirée Sy
Journal of Usability Studies, Volume 2, Issue 3
(the most-cited paper in JUS)
http://www.uxpajournal.org/
80. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017
References and Resources
Albert, Bill, Tom Tullis, and Donna Tedesco. Beyond the Usability Lab.
Gothelf , Jeff. http://blog.usabilla.com/5-effective-ways-for-usability-testing-to-play-nice-with-agile/
Henry, S.L. and Martinson, M. Evaluating for Accessibility, Usability Testing in Diverse Situations. Tutorial, 2003
UPA Conference.
Krug, Steve. Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems.
Patton, Jeff (with Peter Economy). User Story Mapping, Discover the whole story, build the right product.
Rubin, Jeffrey and Dana Chisnell. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective
Tests. John Wiley & Sons, Inc.
Usability.gov
W3C Web Accessibility Initiative - http://www.w3.org/WAI/
UXPA – User Experience Professionals Association - http://www.uxpa.org