Stand-alone UX research generates a lot of data, but without a comparison to other benchmarks, you're often left wondering how your website stacks up in the real world. One of the best ways to put your task scenarios and metrics into context is to see how you compare against the competition. To provide a meaningful comparison, UserZoom presents MeasuringU's Jeff Sauro, who will provide Best Practices for Benchmarking the Website User Experience. View this UserZoom webinar on-demand and discover: TOP solutions, tools and methodologies for Competitive UX Benchmarking HOW to track the most important competitive metrics and stand out from the pack PRACTICAL tips for Competitive UX Benchmarking you won't find anywhere else

1. Best Practices For Benchmarking the
Website User Experience
Jeff Sauro

2. 2015 MEASURING U
RAPIDLY TEST USABILITY & MEASURE UX
UserZoom is the All-in-One UX Research Platform for the Data-Driven Enterprise

3. 2015 MEASURING U
• Chat box is available if you have any questions
• Look out for a quick poll question
• There will be time for Q&A at the end
• We will be recording the webinar for future viewing
• All attendees will receive a copy of the slides/recording
• Continue the discussion using #UZWebinar
Quick Housekeeping

4. 2015 MEASURING U
About MeasuringU
MeasuringU is a UX research firm based in Denver, Colorado USA focusing on the analysis of human
behavior and quantifying the user experience. Since 2004, we've been helping clients answer
questions such as:
• How usable is an application?
• What is the probability a user will purchase a product?
• What are the interaction problems with the mobile application?
• How large of a sample size do we need?
• Which design has a statistically higher conversion rate?
• What are the problems in our website navigation?

5. 2015 MEASURING U
Who we work with
5

6. 2015 MEASURING U
Popular Services: Quant + Qual
6
Measurement Strategy &
Statistical Analysis
Unmoderated Remote
StudiesInformation Architecture/
Findability Testing
Eye-Tracking Studies
Moderated Lab Based
Usability Studies
Mobile Usability Testing

7. 2015 MEASURING U
Why Benchmark?
7
Measure
Before
Measure
After
Make
Changes
Measure, Analyze & Improve
Did improvements make a difference?

8. 2015 MEASURING U
Why Competitive Benchmarking?
8
Makes interpreting the subsequent data easier: the focus becomes less on
absolute scores and more on relative comparisons.
Less of a focus on problems with task scenarios and more on comparison
Answers the question: “Compared to What”
VS.

9. 2015 MEASURING U
UX Benchmarking Process
9
Planning Conducting Analyzing
1. Identify users
2. Finding users
3. Define tasks
4. Comparative vs. Standalone
5. Study metrics
6. Task metrics
7. Sample size
1. Pilot and Program
2. Field the study
1. Summary
2. Study Overview
3. Task Metrics
4. Test Metrics
5. “Why”
6. Click-Paths
2 to 8 Weeks

10. 2015 MEASURING U
10 Best Practices for Competitive Benchmarking
1. Collect Data at the Task and Study Level
2. Measure Website & Brand Attitudes
3. Use Standardized Metrics Where Possible
4. Consider Prior Experience
5. Have Task Success Criteria
6. Decide Between Within- vs Between- Subjects
7. Have a Sufficient Sample Size
8. Measure Preference
9. Compare to Other Reported Measures
10.Rinse & Repeat
10

11. 2015 MEASURING U
1. Collect Data at the Task and Study Level
11
Task Level
• Completion Rates
• Task Time
• Perceived Ease
Study Level
• Overall measure of quality
• Loyalty
• Usability System Usability Scale (SUS)
• Brand Affinity
• Likelihood to purchase
• Preference
Collect a mix of attitudes and actions before and after the tasks

12. 2015 MEASURING U
Task Completion Rates
12
• CODE TASK SUCCESS AS 1
• CODE TASK FAILURE AS 0
• Define a success criteria and do not give
partial credit in the computations

13. 2015 MEASURING U
Task Time
13
• AVERAGE TASK COMPLETION TIME
• MEAN TIME TO FAILURE
• AVERAGE TIME ON TASK

14. 2015 MEASURING U
Task Ease
14
SINGLE EASE QUESTION (SEQ)
Overall, this task was?
1 2 3 4 5 6 7
Very Difficult Very Easy

15. 2015 MEASURING U
2. Measure Website & Brand Attitudes
• Unaided Branding
• When thinking about laptop computers, which brands come to mind?
• What car rental companies can you think of?
• Aided Branding
• Which of the following hotel booking websites are you familiar with?
• Hotels.com
• Blozert.com
• Expedia.com
• Brand Favorability & Overall Attitude
• How would you rate your attitude toward the following brands?
15
HierarchyofEffects

16. 2015 MEASURING U
Interaction of prior attitude on scores
Different experiences for the most and least favorable users
Most favorable users actually get less
favorable -4%
Least favorable users actually get more
favorable +18%
Most Favorable
Least Favorable
Before After

17. 2015 MEASURING U
Measure Attitude Lift
17
Change in attitude “Lift” can be more informative than absolute attitude
4.7 5.35.3 4.5
1
2
3
4
5
6
7
Budget Enterprise
VeryFavorableVeryUnfavorable
12% 15%

18. 2015 MEASURING U
3. Use Standardized Metrics Where Possible
18
SUS SEQ

19. 2015 MEASURING U
SUPR-Q items & Factors
19
USABILITY CREDIBILITY (TRUST & COMFORT)
LOYALTY APPEARANCE
• It is easy to navigate with this website
• This website is easy to use
• I will likely visit the website in the future
• How likely are you to recommend this website to
a friend or colleague? [NPS]
• The information on this website is credible.
• The information on this website is trustworthy.
• The website has a clean and simple presentation
• I found the website to be attractive
Standardized User Experience Percentile Rank-Questionnaire
Sauro, J. (2015) "SUPR-Q: A Comprehensive Measure of the Quality of the Website User Experience" in Journal of Usability
Studies Issue 2, Vol. 10, Feb. 2015, pp. 68-86

20. 2015 MEASURING U
Selection of websites in SUPR-Q database
20
Retail
Amazon
eBay
Zappos
Apple
Target
JC Pennys
Wal-Mart
Container Store
Crate & Barrel
Crumpler
Oriental Trading
Sears
LL-Bean
Michaels
Pier 1 Imports
Macys
Netflix
BestBuy
OfficeMax
OfficeDepot
Airlines
United Airlines
Frontier Airlines
American Airlines
Southwest
Jet Blue
Delta
Frontier
3rd Party Travel
Travelocity
Expedia
Orbitz
Kayak
Government
Illinois
Colorado
California
CDC
USA.gov
New York State
Floral Service
FTD
1-800 Flowers
ProFlowers
3rd Party Automotive
Edmunds
Kelly Blue Book
Cars
Autotrader
Financial Services
PayPal
Fidelity
TD Ameritrade
Vanguard
Bellco
US Bank
Chase
eTrade
Scrottrade
Wells Fargo
News & Information
Wall Street Journal
CNN
FoxNews
New York Times
IMDB
Yelp
Craigslist
Motley Fool
Yahoo
Social Networking
Facebook
Twitter
LinkedIn
Flickr
Relationship/Dating
Match.com
eHarmony
PlentyofFish.com
Hotels
Hyatt
Hilton
Cell Phone Carriers
AT&T Wireless
Sprint
T-Mobile
• 200+ Total Websites
• 10,000 responses
• 30-200 per website

21. 2015 MEASURING U
SUPR-Q “UX Website Quality” 2013
21
88%
91%
99%
89%
90%
98%
98%
87%
79%
60%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Industry Average = 88%

22. 2015 MEASURING U
Systems Usability Scale (SUS)
22
51 2 3 4
Strongly Disagree Strongly Agree
I think that I would like to use this system frequently
I found the system unnecessarily complex
I thought the system was easy to use
I think that I would need the support of a technical person to be able to use this system
I found the various functions in this system were well integrated
I thought there was too much inconsistency in this system
I would imagine that most people would learn to use this system very quickly
I found the system very cumbersome to use
I felt very confident using the system
I needed to learn a lot of things before I could get going with this system

23. 2015 MEASURING U
Raw SUS Scores & Percentile Ranks
23
Average SUS Score is a 68
(500 products)
For Websites, 2 SUPR-Q Usability Items Predict Most (64%+) of SUSs 10 Items

24. 2015 MEASURING U
4. Consider Prior Experience
24
Repeat users rated websites as 11% more usable than first-time users
61 67
50
55
60
65
70
75
80
85
90
First Time User Repeat User
n=764 n=422
SUSScore
11%

25. 2015 MEASURING U
Brand Attitude & Lift by Experience
5.66 6.395.76 6.38
1
2
3
4
5
6
7
Low Experience High Experience
Favorability(LowtoHigh)
Before
After
Once – 5 times 6 times – 10+ times
11%

26. 2015 MEASURING U
Net Promoter Score (NPS) By Site Experience
26
-11%
19% 51% 62%
-2%
21% 62% 70%
-60%
-40%
-20%
0%
20%
40%
60%
80%
100%
0 times 1 to 6 times 6 to 10 times 10+ times
2011
2013
Analysis by Experience, Weight or Statistically Control (Type I ANOVA)

27. 2015 MEASURING U
5. Have Task Success Criteria
27
Spend a few minutes on the website to get to know what it has to offer. Select a product that interests you
and add it to your cart.
Find a toaster that has been reviewed by at least 30 people so you know what to expect when you purchase
it. You also want it to have at least a 4-star rating. You would like to spend no more than $100.00 on your
toaster.
VS
Open:
Open tasks are fine, but closed-ended tasks with a solution often better differentiate between competitors.
Closed:

28. 2015 MEASURING U
6. Within- vs Between- Subjects
28
Factor to Consider Between Within
Sample Size & Power - +
Carryover Effects + -
Impact on Attitudes + -
Comparative Judgment - +
Study Duration + -

29. 2015 MEASURING U
7. Have a Sufficient Sample Size
29
Difference to Detect
90% Confidence & 80% Power
Between-Subjects Sample Size Within-Subjects Sample Size
20% 150 50
10% 614 115
5% 2468 246
4% 3860 312
3% 6866 421
2% 15452 640
1% 61822 1297
See Chapter 6 in Quantifying the User Experience

30. 2015 MEASURING U
Stand Alone Benchmark Sample Size
30
Sample Size Margin of Error (+/-)
10 27%
20 20%
30 17%
40 15%
50 13%
90 10%
115 9%
150 8%
200 7%
260 6%
380 5%
600 4%
1000 3%

31. 2015 MEASURING U
8.Measure Preference
31
87%
13%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
WebsitePreference
For Statistical Significance See Measuringu.com/blog/preference-data.php

32. 2015 MEASURING U
9.Compare to Other Reported Measures
32 Available At Measuringu.com/products.php

33. 2015 MEASURING U
10. Rinse & Repeat
33
TaskCompletionRate
If the ultimate goal of benchmarking is to drive improvement, then measure overtime
73% 71% 64% 67% 76%85% 82% 72% 60% 68%79% 60% 69% 57% 86%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%

34. 2015 MEASURING U
SUPR-Q: 2011 vs 2013
34
Did the Quality of the Website Improve After the Redesign?
77% 73% 86% 72% 41% 29%81% 76% 79% 75% 75% 35%
SUPR-Q Usability Credibility Loyalty Appearance NPS (Raw %)
2011
2013

35. 2015 MEASURING U
Best Practices for Competitive Benchmarking
1. Collect Data at the Task and Study Level
2. Measure Website & Brand Attitudes
3. Use Standardized Metrics Where Possible
4. Consider Prior Experience
5. Have Task Success Criteria
6. Decide Between Within- vs Between- Subjects
7. Have a Sufficient Sample Size
8. Measure Preference
9. Compare to Other Reported Measures
10. Rinse & Repeat
35
@MeasuringU
www.MeasuringU.com
#UZWebinar

36. 2015 MEASURING U36
Questions
Continue the discussion using #UZWebinar

37. 2015 MEASURING U
Need more info?
Get answers about UserZoom. We are here to help.
Talk with one of our knowledgeable specialists.
Contact us
UserZoom
1054 South De Anza Blvd, Suite 102
San Jose, CA 95129
Toll free number: (866) 599 1550
www.linkedin.com/company/userzoom@UserZoomFollow us