A half day course on Interaction Design taught by Mark Billinghurst from the HIT Lab NZ at the University of Canterbury

1. An Introduction to
Interaction Design
g
Mark Billinghurst
HIT Lab NZ

2. Mark
PhD Electrical Engineering
University of Washington
Interaction Design
Museum experiences
Tools for designers
Augmented Reality
Mobile AR, Evaluation
AR Evaluation,
Multimodal Interfaces, Collaborative
Collaboration
Enhanced FtF and remote collaboration
Social networking

3. Workshop Overview
1. What is Interaction Design
2.
2 Understanding Interaction
3. Understanding User Experience
4. Design and Prototyping
5.
5 Evaluation
6. Design in the Real World
7. Resources

4. Source Material
Interaction Design: Beyond Human
p
Computer-Interaction
Rogers, Sharp, Preece
http://www.id-book.com/
htt // id b k /

5. 1. What is Interaction Design?
Mark Billinghurst
Preece Chapter 1

6. What to do?
Imagine
I i
You’re bringing a new product to market
Your #2 competitor has been in the market for
over a year, selling millions of units
y g
Your #1 competitor launches the same month
Your technology is slower than your competitors
Your technology is older than your competitors
Your last product failed in the market

7. Do you compete on Price ?
Do you compete on Technology ?
Do you compete on Features ?
Wrong: Compete on user experience !

9. Nintendo Wii
Cheap - $500
Unique game play
Wireless 3 DOF controller
Position and orientation sensing
Aiming to broaden user base
g
Can play previous games/downloads

11. Sales to Sept 2011

12. “The
“Th product is no longer
d i l
the b i f l The
th basis of value. Th
experience is ”
is.
Venkat Ramaswamy
The Future of Competition.

13. Gilmore + Pine: Experience Economy
Gil Pi E i E
experiences Emotion
E ti
services
products
components Function
Sony CSL © 2004

14. The Value of Experience
50c
50
$3.50
20c

15. Good Experience Design
Reactrix
Top down projection
Camera b d input
C based i
Reactive Graphics
No instructions
No training

18. Using the N-gage

19. SideTalking
www.sidetalkin.com

22. Interaction Design
“Designing interactive products to support people
in their everyday and working lives”
lives
Preece, J., (2002). Interaction Design
Design of User Experience with Technology

23. Goals of interaction design
Develop usable products
Usability means easy to learn, effective to use and
y y ,
provide an enjoyable experience
Involve users in the design process

24. Bill Verplank
Interaction Design involves answering three questions:
What do you do? - How do you affect the world?
What do you feel? – What do you sense of the world?
What do you know? – What do you learn?
y y

26. HCI and interaction design
26
www.id-book.com

28. Interaction Design Process
I D P
Identify needs/
establish
requirements
q
(Re)Design
Evaluate
Build an
interactive
version
Develop alternative prototypes/concepts and compare them Final product
And iterate, iterate, iterate....
iterate iterate iterate

29. 2. Understanding Interaction
Mark Billinghurst

30. Understanding the problem space
What do you want to create?
y
What are your assumptions?
What are your claims?
Will it achieve what you hope it will? If so, how?

31. Analysing the problem space
Are there problems with an existing product or
user experience?
p
Why do you think there are problems?
How do you think your proposed design ideas
might overcome these?
When designing for a new user experience how
will the proposed design extend or change
current ways of doing things?
y g g

32. Conceptual model
A conceptual model i
t l d l is:
“a high-level description of how a system is organized and
operates.
operates.” (Johnson and Henderson, 2002, p. 26)
Need to first think about how the system will appear
to users (i.e. how they will understand it)
Not a description of the user interface but a structure
outlining the concepts and relationships between them
Benefits
B f
Enables “designers to straighten out their thinking before they start
laying out their widgets” (p. 28)
y g g (p )
Provides a working strategy and a framework of general concepts and
their interrelations

33. Interface metaphors
Designed be i il
D i d to b similar to a physical entity b also has own
h i l i but l h
properties
e.g.
e g desktop metaphor, search engine
metaphor
Exploit user’s familiar knowledge, helping them to understand
‘the unfamiliar’
Conjures up the essence of the unfamiliar activity, enabling
users to leverage of this to understand more aspects of the
unfamiliar f ti lit
f ili functionality
People find it easier to learn and talk about what they are
doing at the computer interface in terms familiar to them

34. Example: The spreadsheet
Analogous to ledger
sheet
Interactive and
computational
p
Easy to understand
Greatly extending
what accountants
and others could do
www.bricklin.com/history/refcards.htm

35. Why was it so good?
It
I was simple, clear, and obvious to the users how to
l l d b h h
use the application and what it could do
“it i j
“i is just a tool to allow others to work out their
l ll h k h i
ideas and reduce the tedium of repeating the same
calculations.
calculations ”
capitalized on user’s familiarity with ledger sheets
Got h
G the computer to perform a range of different
f f diff
calculations in response to user input

36. Another classic
8010 S office system targeted at workers not
Star ff d k
interested in computing per se
Spent several person-years at beginning working out
S l b i i ki
the conceptual model
Simplified h l
Si lifi d the electronic world, making it seem more
i ld ki i
familiar, less alien, and easier to learn
Johnson et al (1989)

38. Benefits of interface metaphors
Makes l
M k learning new systems easier
Helps users understand the underlying
p y g
conceptual model
Can be innovative and enable the realm of
computers and their applications to be made
more accessible to a greater diversity of users

39. Problems with interface metaphors
P bl h f h
(Nielson,
(Nielson 1990)
Break conventional and cultural rules
e.g., recycle bin placed on desktop
Can constrain designers in the way they conceptualize a problem
Conflict with design principles
Forces users to only understand the system in terms of the
metaphor
Designers can inadvertently use bad existing designs and transfer
the bad parts over
h b d
Limits designers’ imagination with new conceptual models

40. Microsoft Bob

41. Summary points
Need to have a good understanding of the problem space
specifying what it is you are doing, why, and how it will support
p y g y g, y, pp
users in the way intended
A conceptual model is a high-level description of a product
what users can do with it and the concepts they need to
understand how to interact with it
Decisions about conceptual d i should be made before
D ii b l design h ld b d b f
commencing any physical design
Interface metaphors are commonly used as part of a
conceptual model

42. 3. Understanding User Experience
Mark Billinghurst

43. What is a Bad Experience
?

44. Misleading Graphics

45. Cognitive Overload for Simple Task
g p

46. Cognitive Overload for Simple Task

47. Dangerously Undermining Task
Many p p in Palm Beach, Florida that wanted to vote for
y people
Al Gore, accidentally voted for Pat Buchanan
Estimated 4000 voted
“wrong“
wrong
Another 19,000
punched both holes
(invalid vote)

48. Interaction Design is User Centered
A methodology for ensuring good user
experiences with products by getting
feedback from users to inform the design.

49. 1. User research and needs analysis
Identify needs/
establish
requirements
i t
(Re)Design
Evaluate
Build an
interactive
version
Final p oduct
a product

50. What, How and Why?
•Why:
Requirements definition: the stage
where failure occurs most
h f il t
commonly
Getting requirements right is crucial
50 www.id-book.com

52. What, how and why?
•What
1. Understand users, task, context
2. Produce a stable set of requirements
•How:
Data th i
D t gathering activities
ti iti
Data analysis activities Iterate
Expression as ‘requirements’
requirements
•Why:
y
Requirements definition: failure occurs most commonly
Getting requirements right is crucial

53. What
Wh are the Requirements?
h R ?
Functional
What should the system do? What has it done?
Data
Environment or Context of use
Physical: dusty? noisy? light? heat? humidity? ….
Social: sharing of files of displays work alone ..
files, displays,
Organizational: IT support, hierarchy, training..
Users: Wh are they?
U Who h ?
Usability: learnability, throughput, flexibility..

54. Who are your Users?
Everyone!
y

55. Who REALLY are your Users/Stakeholders?
•Not as obvious as you think:
— those who interact directly with the product
— those who manage direct users
— those who receive output from the product
— those who make the purchasing decision
— those who use competitor s products
competitor’s
•Three categories of user (Eason, 1987):
— primary: frequent hands-on
p y q
— secondary: occasional or via someone else
— tertiary: affected by its introduction, or will influence its purchase

56. Who are the Stakeholders?
Check-out operators
• Suppliers
• Local shop
owners
Managers and owners Customers
C t

57. Understanding Specific Needs

58. Designing for Everyone
Designing for Everyone pleases No one

59. Consider the Whole User

60. Data Gathering Techniques (1)
Questionnaires
Looking for specific information
Qualitative and quantitative results
Good for getting data from a large dispersed group
large,
Interviews
Good for exploring issues, using props
Structured, unstructured or semi-structured
But are time consuming and difficult to visit everyone

61. Data Gathering Techniques (2)
Workshops or focus groups
W kh f
Group interviews/activities
Good
G d at gaining a consensus view and/or
i i i d/
highlighting areas of conflict
Observations
Spending time with users in day to day tasks
Good for understanding task context
requires time and can result in a huge amount of data
Documentation
Procedures and rules written down in manuals

62. A day in the Life of.. Cultural Probes.. Role Playing..

63. Case Study: Equator Domestic Probes

64. Some Basic Guidelines
• Focus on identifying the stakeholders’ needs
• Involve all the stakeholder groups
• Involve more than one person from each group
• Use a combination of data gathering techniques
•S
Support the process with props such as prototypes
h ih h
• Consider carefully how to record the data
y

65. User Interactions =The Golden Egg

66. Why?
Users can give you lots of f db k about what your
U l f feedback b h
product does well and what it doesn’t do well.
Users can give you insight into their lives and needs
helping
h l i you understand the context in which your
d d h i hi h
product will be used
Users can inspire design solutions you haven’t
thought of.
h h f

67. Interviewing Techniques
Good interviewing is a skill and needs to be
done properly to ensure y maximize the
p p y you
opportunity you have with your users
Tips for interacting with end users:
1. Listen
2. Watch
3. C t T t
3 Create Trust
4. Inform Designg

68. 1. Listen
Most important part of interviewing.
You are not there to train the user or to
demonstrate how much you know. You
are interviewing an expert to gain knowledge.
Treat them like a precious partner and remember
they know a lot more about their work then you do.

69. 2. Watch
Remember users will tend to want to say what
they think you want to hear.
Create opportunities to observe users rather than
ask users.

70. Case Study –
A usability professional is interviewing a user:
Professional: “Do you know how to set the margins?”
User: “Oh yes, I do that all the time.”
Professional: “Could you show me how to do it?”
User: “Sure.” (user presses a series of buttons unrelated
to setting margins – the button sequence is actually
changing a different setting).
“See it beeped so the margins are set ”
See set.

71. 3. Create Trust
Users will be nervous that they will appear
stupid or incompetent.
“We are testing design, not you”
To get good data, user must feel relaxed
and trusting.

72. A user’s perspective:
Well okay, today’s the day. I have to report to some building on 14th street. I must
admit I m a bit nervous. When I spoke to the woman on the phone, she asked
I’m
me a whole lot of questions about my background and experience. She
seemed particularly gleeful that I wasn’t competent using computers and
equipment. I’m g
q p glad she is happy but for me it’s a recurring p
ppy g problem.
I’ve always felt intimidated with electronics. She wants me to use something on
the computer while some people watch me. Well, it’s an easy $50 bucks and
seeing that I don’t know the people, it can’t be too embarrassing…
On the other hand, what if I’m the first person in the world that doesn’t understand
how to do whatever I’m supposed to do? What if I totally bomb? What if they
ask me a question that is embarrassing and they find out how stupid I really
am. Well, I’ll give it a go this time but I don’t think I can do this again.

73. 4. Inform Design
g
User research does not dictate your design but
rather informs you so that you design better.
y y g

74. Woodblock Study Example
y p
Users were asked to place
stickers representing f
ti k ti functions
ti
On a block model.
Resulting design did not copy word
for word where the users placed
buttons.
Resulting design was informed by
how users grouped buttons and
by observation of users interacting
with the stickers.

75. Summary
Four basic activities in the design process
1. Establishing requirements
2. Designing alternatives
3. Prototyping
4.
4 Evaluating
User-centered design rests on three principles
1.
1 Early focus on users and tasks
E l f dt k
2. Empirical measurement using quantifiable & measurable
usability criteria
3. Iterative design
75 www.id-book.com

76. 4. Design and Prototyping
Mark Billinghurst

77. 2. (Re)Design
2 (R )D
Identify needs/
establish
requirements
q
(Re)Design
Evaluate
Build an
interactive
version
Final product

78. Design Inspiration

79. Discovering Unmet Needs
Digital Cameras for Apple
Late 80s – What would happen if we married
a computer with a digital camera?
Many studies of current photography use.
Uncovering of unmet needs in traditional experience
- not enough photos
- not knowing whether photo good or not
- ability t add sound
bilit to dd d
- ability to organize
Features determined through research and designed
Iteratively.

80. Brainstorming
Best with interdisciplinary team

81. Other Products
Notice all the iPod look-alikes?

82. Tools for Effective Design
Personas
Scenarios
Storyboards (comic strip, movie technique)
Wireframes and Mock-ups
yp
Prototypes

83. Persona Technique
P T h
Personas are a design tool to help visualize who
you are designing for and imagine how this
person will use the product
A persona is an archetype that represents the
behavior and goals of a group of users
Based on insights and observations from
customer research

84. How to Develop a Persona
Brainstorming
Imagining different people using the technology
Design Research
Ethnographic Studies
Contextual Inquiry
Field Studies
Usability Studies
Marketing Research
Focus Groups
Sales Feedback

85. Gunther the Ad Guy
Gunther is from Germany. He
Travels extensively for work and
As he is an advertising executive
he needs to present concepts to
clients quickly and easily. He is
a person very well-versed in new
technologies
t h l i and wishes h had
d i h he h d
easier portable solutions for his
presentations…..

86. 86 www.id-book.com

87. How to use your Personas
Use them to find real people for usability testing.
Use them as a foundation to discuss any design
issues/feature issues that come up.
/f h
Use them to evaluate competitor’s products.
Use them to keep you honest.
Keep updating them as part of your continued user
research.h
Use them to find real customers you can have on-
going relationships with
with.

88. Scenarios
Usage Scenarios are narrative descriptions of how
the product meets the needs of a persona
Short (2 pages max)
Focus on unmet needs of persona
Concrete story
C
Set of stories around essential tasks, problems...
Use to test ideas

89. Scenario Technique
Develop with personas
Prioritize experiences to support
Focus on top three experiences but support other
experiences

90. A business woman travels from SF to Paris on a business trip. On her
way to the airport she narrowly misses a traffic delay She avoids
delay.
the jam because her Smartphone beeps and send her a warning
text message on her route from the office to the airport.
Upon arrival the location-sensitive Smartphone notifies the airline that
she‘ll check in shortly and an airline employee finds her
immediately and takes her baggage Her display shows that her
baggage.
flight is on time and provides a map to her gate. On the way she
downloads tourist information (maps, events) for Paris.
Symbian

91. Once found her seat, she begins t review th d
O f dh t h b i to i the downloaded
l d d
information. She books a ticket for an opera she wants to see. Her
Smartphone makes the booking using her credit card number
p g g
stored in memory. The security software of the Smartphone
protects her against fraud.
The Smartphone stores the opera booking along with emails written
ith ritten
on the plane. As soon as she steps off the plane, it makes the calls
and sends the emails. As she leaves the airport, a map appears on
the display and guides her to her hotel
Symbian

92. Storyboarding
Sequence of sketches showing use of system in
everyday use context
Concrete example
Easier (faster) to grasp than text based stories
Means of communication with users and system
developers
p
Sketches, not drawings...
Use to test interaction and make sure design works

93. Turning Scenarios into Storyboards
1. Photographs to match personas
(can be used to communicate environment or
profession)
2. Drawings
3. Draw-over acted photos
4. Detail not important – communicate the experience
rather than specifics
specifics.

94. Example Storyboard

95. Example Storyboard p.2

96. Wireframes & Mock Ups
Roughly sketched sequences exploring and validating
interaction design.
Focus is on navigation and functionality.
Usually explores one interaction but can also be used
as the foundation for development of testing
p
prototypes.
yp

99. Prototyping
Mark Billinghurst

100. Design and Prototyping
D dP
Identify needs/
establish
requirements
q
(Re)Design
Evaluate
Build an
interactive
version
Final product

101. Prototypes Redefined
Prototyping is common activity in product
development.
Traditional prototypes can take weeks or months.
RAPID prototyping is a design technique
to speed up d l
d development of ideas.
f d
A rapid prototype can take as little as a few minutes
to make.

102. Benefits of RAPID Prototyping
Fast and inexpensive
Identifies problems before they re coded
they’re
Elicits more and better feedback from users
Helps developers think creatively
Gets users and other stakeholders involved
early i th process
l in the
Fosters teamwork and communication
Avoids i i
A id opinion wars
Helps decide design directions

103. Types of Prototypes
Low Fidelity – quick and dirty, easy access materials
like cardboard and paper.
High Fidelity – more involved electronic versions
similar in materials to final product.

104. Paper Prototyping (Low Fidelity)
Quick and simple means of sketching interfaces
Use ffi
U office materials
t i l
Easier to criticize, quick to change
Creative process (develop in team)
Can also use for usability test (focus on flow of
interaction rather than visuals)
Used a lot to test out concepts before real design
begins.
g

105. Paper Proto: Create Widgets

106. Paper Prototyping w/Hardware

107. Blank Model Prototypes

108. Screen M kU Prototypes
S MockUp P

109. Interactive Role Play

110. Interactive Role Play in Environment
IDEO.com

111. Half-functional prototypes
More hi-fi
HTML ((or
Powerpoint) ...
Mostly precoded
flow, but allows
clicking and
experiencing flow

112. Wireframing T l
Wi f i Tools
Pidoco (websites mobile, desktop)
(websites, mobile
No programming
http://www.pidoco.com
WireframeSketcher
Plug-in for Eclipse
http://wireframesketcher.com/
htt // i f k t h /
Omnigraffle (Mac)
Digramming tool
http://www.omnigroup.com/products/omnigraffle/
http://www omnigroup com/products/omnigraffle/

113. Mobile Rapid Prototype Tools
Android: App Inventor
http://appinventor.googlelabs.com/about/
iPhone – Meta collection
http://iphoneized.com/2009/11/21-prototyping-
mockup-wireframing-tools-iphone-app-development/
mockup wireframing tools iphone app development/

114. Tethered Prototypes (High Fidelity)
Goal:
Communicate Vision
HIGH FIDELITY
PROTOTYE

115. Rapid Prototyping
Speed development time with quick hardware mockups
handheld device connected to PC
LCD screen USB phone keypad, Camera
screen, keypad
Can use PC development tools for rapid development
Flash, Vi l Basic,
Fl h Visual B i etc

116. ‘Wizard-of-Oz’ Prototyping
• The user thinks they are interacting with a
computer, but a developer is responding to
output rather than the system.
• Usually done early in design to understand
users’ expectations
User
>Blurb blurb
>Do this
>Why?
116 www.id-book.com

117. Typical Iterations for HW/SW Product:
1.
1 Paper prototype to evaluate conceptual model
2. Interactive computer-based prototype with rough
screens to evaluate feature placement
p
3. Tethered prototype to evaluate button + screen
interactions
4. Real device prototyped with major features working
5. Real device prototyped with all features working and
graphic design implemented

118. Design and Prototyping
D dP
Identify needs/
establish
requirements
q
(Re)Design
Evaluate
Build an
interactive
version
Final product

119. Design Case Study

120. ITERATIVE DESIGN PROCESS
Design
(redesign)
Prototype
Evaluate Test Design with Users
(Fix Issues)
( i ) (Note problems)

121. MOBILE AUGMENTED
REALITY FOR SPATIAL
NAVIGATION
Sharon Brosnan
0651869
Bachelor of Science in Digital Media Design

122. BUNRATTY FOLK
PARK Sharon Brosnan
0651869
Bachelor of Science in Digital Media Design

123. BUNRATTY FOLK PARK
Irish visitor attraction run by Shannon
Heritageg
19th
19 h century life is recreated
lf d
Buildings from the mid-west have been
relocated t th 26-land surrounding Bunratty
l t d to the 26 l d di B tt
Castle
30 buildings are set in a rural or village setting

124. AUGMENTED REALITY
In Bunratty Folk Park:
Allows the visitor to point a camera at an
exhibit, the device recognises its by it’s
location and layers digital information on to
f
the display
p y
3- dimensional virtual objects can be
positioned with real ones on display
Leads to dynamic combination of a live camera
view and information

125. NAVIGATIONAL AID
Smartphone Platform
Most people carry mobile phones and are
comfortable with them
Ideal A
Id l Augmented Reality Technology
d R li T h l
Global tracking tools
g
Wireless communication capabilities
Location based computing
Large display for interaction

126. DESIGNING
FOR THE VISITOR

127. HUMAN CENTRED DESIGN
Goal f the Navigational Aid
G l of th N i ti l
Easy to use, clear and understandable
Useful to visitors
Creating interaction between the visitor and the aid through
the user interface
Engage the visitor
E h ii
To ensure this…
It is necessary to understand the visitor of a navigational aid in
Bunratty Folk Park
Identify visitor motives and goals while going through the Folk
Park.
P k

128. HUMAN CENTRED DESIGN
Understanding the technology and Related
Works
Literature
Similar Technologies
g
Electronic Tours in Museum Settings
Interactions design in Outdoor Museums
g
Understanding the User over time
Observations
Ob
Interviews
te v ews

129. HUMAN CENTRED DESIGN
Findings
Most visitors do not use the map
Most visitors have mobile phones
Visitors want more information
View the Folk Park at their own pace
Information should be straight to the point
Large social interaction within groups
g g p

130. NEXT STEPS FROM RESEARCH
Define Visitors Goals
Define Functionalities f the
D fi F ti liti of th Aid
Develop Personas – visitors who use the Aid
Develop Scenarios – how the persona uses the
p p
navigational aid in the Folk Park
Draw up Storyboards on scenarios

131. FUNCTIONALITY
View Options Augmented Reality
Camera View Features for
Map View navigation
List View Text Information
3D Objects
Sub Options
Sub-Options 3D Tour Guide
Places 3D Placement of
Events Buildings
g
Restaurants

132. STORYBOARD

135. ITERATIVE DESIGN

136. ITERATIVE DESIGN PROCESS
Prototyping and User Testing
Low Fidelity Prototyping
Sketches
Paper Prototyping
Post-It Prototyping
PowerPoint Prototyping
High Fidelity Prototyping
Wikitude

137. INITIAL SKETCHES
Pros:
• Good for idea generation
• Cheap
• Concepts seem feasible
Cons:
• Not great feedback gained
• Photoshop not fast enough for
Photoshop not fast enough for
making changes

138. POST IT PROTOTYPING
Camera View with 3D
Camera View with 3D
First Draft Second Draft Third Draft
• Selection
Selection • Home button added
Home button added
highlighted in blue for easy navigation to
main menu

139. POWERPOINT PROTOTYPING
Benefits
Benefits
• Used for User Testing
• Interactive
• Functionalities work when following the
Functionalities work when following the
story of Scenario 1
• Quick
•E
Easy arrangement of slides
t f lid
User Testing
• Participants found
• 15 minute sessions screen captured
• ‘Talk Allowed’ technique used
Talk Allowed technique used
• Notes taken
• Post Interview
Post‐Interview

140. WIKITUDE
Popular augmented reality
browser for mobile devices
Mapping
Point of Interest abilities
Multiplatform
M li l f
Shows the points of interest
of Bunratty Folk Park
Markers can be selected in and an
information pop-up appears

141. WIKITUDE
User Testing
Application well received
Understandable
Participants playful with the technology

142. FINAL CONCEPT
DESIGN

143. FINAL DESIGN CONCEPT
Key Issues
K I
Fix issues found in previous sessions
Design with guidelines in mind
Appealing to the Mental Model
Icon Design
Aesthetic Design
Colour/Font
Buttons
Look

144. VIDEO PROTOTYPE
Flexible tool for capturing the use
of an interface
Elaborate simulation of how the
navigational aid will work
Does not need to be realistic in
every detail
every detail
Gives a good idea of how the
finished system will work
finished system will work

145. 5. User Evaluation
Mark Billinghurst

146. 4. Evaluation
Identify needs/
establish
requirements
i t
(Re)Design
Evaluate
Build an
interactive
version
Final product

147. What is evaluation?
Evaluation is concerned with
gathering d about the usability of
h i data b h bili f
a design or product by a specified
group of users for a particular activity
within a specified environment or work
context

148. When to evaluate?
Once the product has been developed
p
pros : rapid development, small evaluation cost
p p ,
cons : rectifying problems
redesign &
design implementation evaluation
reimplementation
During design and development
pros : find and rectify problems early
fi d d if bl l
cons : higher evaluation cost, longer development
design implementation

149. Four evaluation paradigms
‘quick a dirty’
qu c and ty
usability testing (lab studies)
field studies
predictive evaluation

150. Quick and dirty
‘quick & dirty’ evaluation: informal feedback
from users or consultants to confirm that their
ideas are in-line with users’ needs and are liked.
Quick & dirty evaluations are done any time.
Emphasis is on fast input to the design process
rather than carefully documented findings.

151. Usability Testing
Recording typical users’ performance on typical tasks in
controlled settings. Field observations may be used.
g y
As the users perform these tasks they are watched & recorded
on video & their key presses are logged.
This data is used to calculate performance times, errors & help
explain why the users did what they did.
User satisfaction questionnaires & interviews are used to elicit
users’ opinions.

152. Laboratory-based studies
Laboratory-based studies
can be used for evaluating the design, or the
implemented system
are carried out in an interruption free usability lab
interruption-free
can accurately record some work situations
some studies are only possible in a lab environment
some tasks can be adequately performed in a lab
q yp
are useful for comparing different designs in a
controlled context

153. Laboratory-based studies
Controlled, instrumented environment

154. Portable equipment for the field
154 www.id-book.com

155. Field Studies
Field studies are done in natural settings
The aim is to understand what users do naturally and
y
how technology impacts them.
In product design field studies can be used to:
- identify opportunities for new technology
- determine design requirements
- decide how to introduce new technology
- evaluate technology in use
use.

156. Predictive Evaluation
Experts apply their knowledge of typical
users,
users often guided by heuristics to predict
heuristics,
usability problems.
Can involve theoretically based models.
A key feature of predictive evaluation is
that users need not be present
Relatively quick and inexpensive

157. Characteristics of approaches
Usability Field Predictive
testing studies
Users do task natural not involved
Location controlled natural anywhere
When prototype early prototype
Data quantitative qualitative problems
Feed back measures & descriptions problems
errors
Type applied naturalistic expert

158. Evaluation approaches and methods
Method Usability Field Predictive
testing studies
Observing x x
Asking x x
users
Asking x x
experts
Testing x
Modeling x

159. Living labs
People’s use of technology in their everyday
lives can be evaluated in living labs.
g
Such evaluations are too difficult to do in a
usability lab
lab.
Eg the Aware Home was embedded with a
complex network of sensors and audio/video
recording devices (Abowd et al., 2000).
159 www.id-book.com

160. User Evaluation Process
160 www.id-book.com

161. DECIDE:
A framework to guide evaluation
- Determine the goals the evaluation addresses.
- Explore the specific questions to be answered.
- Choose the evaluation p di and t h i
Ch th l ti paradigm d techniques
- Identify the practical issues.
- Decide how to deal with the ethical issues.
- Evaluate, interpret and present the data.

162. DECIDE Framework
Determine Goals:
D G l
What are the high-level goals of the evaluation?
How wants the evaluation and why?
H t th l ti d h ?
Explore the Questions:
Create well defined, relevant questions
Choose the Evaluation Paradigm
Influences the techniques used, how data is analyzed
Identify Practical Issues
How to select users, stay on budget & schedule
How to find evaluators, select equipment

163. DECIDE Framework
Decide on Ethical Issues
Informed consent form
Participants have a right to:
-kknow th goals of th study and what will h
the l f the t d d h t ill happen to the fi di
t th findings
- privacy of personal information
Evaluate, Interpret and Present Data
, p
- Reliability: can the study be replicated?
- Validity: is it measuring what you thought?
y g y g
- Biases: is the process creating biases?
- Scope: can the findings be generalized?
- E l i l validity: is the environment influencing the results?
Ecological lidit i th i t i fl i th lt ?

164. Key
K points
• Many issues to consider before conducting an
evaluation study.
• These include: goals of the study; involvment or not
of users; the methods to use; practical & ethical
issues; how data will be collected analyzed &
collected,
presented.
• Th DECIDE f
The framework provides a useful checklist
k id f l h kli
for planning an evaluation study.
164 www.id-book.com

165. Usability Testing

166. Usability Testing
U bl T
Recording performance of typical users doing typical tasks
tasks.
Controlled settings.
Users are observed and timed.
U b d d d
Data is recorded on video & key presses are logged.
The data is used to calculate performance times, and to
identify & explain errors.
User satisfaction is evaluated using questionnaires &
interviews.
Field observations can provide contextual understanding.
166 www.id-book.com

167. Experiments & Usability Testing
Experiments test hypotheses to discover new
knowledge by investigating the relationship between
g y g g p
two or more things – i.e., variables.
Usability testing is applied experimentation.
y g pp p
Developers check that the system is usable by the
intended user population for their tasks.
p p
Experiments may also be done in usability testing.
167 www.id-book.com

168. Usability Testing Research
U bilit T ti & R h
Usability testing
y g Experiments for
p
research
Improve products Discover knowledge
Few participants Many participants
Results inform design Results validated
Usually
U ll not completely
l l statistically
replicable Must be replicable
Conditions controlled as Strongly controlled
much as possible conditions
Procedure planned
p Experimental design
p g
Results reported to Scientific report to
developers scientific community
168 www.id-book.com

169. Pilot Studies
A small trial run of the main study.
Can identify majority of issues with interface design
Pilot studies check:
- that the evaluation plan is viable
- you can conduct the procedure
- that i
h interview scripts, questionnaires,
i i i i
experiments, etc. work appropriately
Iron out problems before doing the main study.

170. Controlled experiments
Designer of a controlled experiment should
carefully consider
y
proposed hypothesis
selected subjects
measured variables
experimental methods
data collection
data analysis

171. Subjects
The h
Th choice of subjects is critical to the validity of
f b l h ld f
the results of an experiment
subjects group should be representative of the
expected user population
In selecting the subjects it is important to
consider things such as their
g
age group, education, skills, culture
The sample size should be large enough (10+) to
be statistically representative of the population

172. Variables
V i bl
Experiments manipulate and measure variables
under controlled conditions
There are two types of variables
independent: variables that are manipulated to
create different experimental conditions
- e g number of items in menus, colour of the icons
e.g. menus
dependent: variables that are measured to find out
the effects of changing the independent variables
- e.g. speed of menu selection, speed of locating icons

173. “Other” Variables
Control variables
e.g. room light, noise…
if controlled => less external validity
Random variables (not controlled)
( )
e.g. fatigue
more influence of random variable => less internal validity
y
Confounding variables
practice
previous experience

174. Hypothesis
A hypothesis is a prediction of the
outcome
what will happen to the dependent variables when
the independent variables are changed
to show that the prediction is right
- dependant variables don’t change by changing
the independent variables
- rejecting the null hypothesis (H0 )

175. Experimental methods
It is important to select the right experimental
p
method so that the results of the experiment
can be generalized
There are mainly two experimental methods
between-groups: each subject is assigned to one
experimental condition
within-groups: each subject performs under all
the different conditions

176. Experimental methods
Between-groups
g p Within-groups
g p
Subjects Subjects
Randomly Randomly
assigned assigned
rimental tasks
rimental tasks
rimental tasks
Condition Condition Condition
erimental task
1 2 3
Condition Condition Condition Condition Condition Condition
1 2 3 2 1 1
Expe
Exper
Exper
Exper
Condition Condition Condition
3 3 2
data data data data data data
Statistical data analysis Statistical data analysis

177. Within vs. Between Subjects
between subjects design
each participant is tested on only one level/condition
p p y
a separate group of participants is used for each
condition
- one group uses HMD other group uses Handheld device
within subjects d i
i hi bj design
participant is tested on each level/condition
- e.g. participants use Handheld device and HMD
repeated measurement
p

178. Between Subjects
Sometimes a factor must be between subjects
e.g. gender, age, experience
Between subjects advantage:
avoids interference effects (e.g. practice / learning effect)
( g p g )
Between subjects disadvantage:
Increased variability = need more subjects
Important: randomised assignment to conditions

179. Within Subjects
Sometimes a f t must be within subjects
S ti factor t b ithi bj t
e.g. measuring learning effects
Within subjects advantages
less participants needed (all p
p p ( participants in all conditions)
p )
differences (variability) between subjects the same across test
conditions
Counterbalance order of presenting conditions
A => B => C B => C => A C => A => B
The order is best governed by a Latin Square

180. Latin Square Design
each condition occurs once in each row and
column
Note: In a balanced Latin Square each condition
both
b th precedes and f ll
d d follows each other condition
h th diti
an equal number of times

181. Data collection and analysis
The choice of a method is dependent on the
type of data that needs to be collected
In order to test a hypothesis the data has to
be analysed using a statistical method
The choice of a statistical method depends on
the type of collected data
h f ll dd
All the decisions about an experiment should
p
be made before it is carried out

182. Observe and Measure
Observations are gathered…
manually (
y (human observers) )
automatically (computers, software, cameras,
sensors, etc )
sensors etc.)
A measurement is a recorded observation
Objective metrics
Subjective metrics

183. Typical objective metrics
task completion time
k l i i
errors (number, percent,…)
percent of task completed
ratio of successes to failures
number of repetitions
number of commands used
number of failed commands
physiological data (heart rate,…)
…

184. Typical subjective metrics
user satisfaction
subjective performance
ratings
ease of use
intuitiveness
judgments
…

185. Data Types
Subjective
Subjective survey How easy was the task
- Likert Scale, condition rankings
Scale 1 2 3 4 5
Not very easy Very easy
Observations
- Think Aloud
Interview responses
Objective
Performance measures
- Time accuracy, errors
Time, accuracy
Process measures
- Video/audio analysis

186. Experimental Measures
E erimental Meas res
Measure What does it tell us? How is it measured?
Timings Performance Via a stopwatch, or
automatically by the device.
Errors Performance, Particular sticking points in a task By success in completing the task
correctly. Through experimenter
observation, examining the route
walked.
Perceived Workload Effort invested. User satisfaction Through NASA TLX scales and other
questionnaires.
i i
Distance traveled and route Depending on the application, these can be used Using a pedometer, GPS or other
taken to pinpoint errors and to indicate performance location-sensing system. By
experimenter observation.
Percentage preferred walking Performance By finding average walking speed,
speed which is compared with normal walking
speed.
Comfort User satisfaction. Device acceptability Comfort Rating Scale and other
questionnaires.
User comments and User satisfaction and preferences. Particular Through questionnaires, interviews and
preferences sticking points in a task. think alouds.
think-alouds.
Experimenter observations Different aspects, depending on the experimenter Through observation and note-taking
and on the observations

187. Statistical Analysis
Once data is collected statistics can be used for analysis
Typical Statistical Techniques
yp q
Comparing between two results
- Unpaired T-Test (for between subjects – assumes normal distribution,
interval scale, homogeneity of variances)
- Paired T-Test (for within subjects – assumes normal distribution, etc.)
- Mann Whitney U test (between subjects – if assumptions are not met)
Mann–Whitney U-test
Comparing between > two results
- Analysis of Variance – ANOVA
- Followed by post-hoc analysis – Bonferroni adjustment
- Kruskal–Wallis (does not assume normal distribution)

188. Case Study: A Wearable Information Space
Head Stabilized Body Stabilized
An
A AR i t f
interface provides spatial audio and visual cues
id ti l di d i l
Does a spatial interface aid performance?
–Task time / accuracy

189. Task Performance
Task
T k
find target icons on 8 pages
remember information space
b i f i
Conditions
A - head-stabilized pages
B - cylindrical display with trackball
C - cylindrical display with head tracking
Subjects
Within subjects (need fewer subjects)
12 subjects used

190. Experimental Measures
Objective
Ob
spatial ability (pre-test)
Many
time to perform task
information recall Different
workload (NASA TLX)
kl d Measures
Subjective
j
Post Experiment Survey
- rank conditions (forced choice)
( )
- Likert Scale Questions
• “How intuitive was the interface to use?”

191. Post Experiment Survey
For each of these conditions please answer:
1) How easy was it to find the target?
1 2 3 4 5 6 7
1=not very easy 7=very easy
For the head stabilised condition (A):
For the cylindrical condition with mouse input (B):
For the head tracked condition (C):
Rank all the conditions in order on a scale of one to three
1) Which condition was easiest to find target (1 = easiest, 3 = hardest)
A: B: C:

192. Results
Body St bili ti Improved P f
B d Stabilization I d Performance
search times significantly faster (One factor ANOVA)
Head Tracking Improved Information recall
no difference between trackball and stack case
Head tracking involved more physical work

193. Subjective Impressions
5
4.5
4
3.5
3
Find Target
2.5
Enjoyable
2
1.5
15
1
0.5
0
A B C
Subjects Felt Spatialized Conditions (
j p (ANOVA):
)
More enjoyable
Easier to find target

194. Subjective Impressions
3
2.5
2
Easiest
1.5 Understanding
Intuitive
1
0.5
0
A B C
Subject Rankings (Kruskal-Wallis)
Spatialized easier to use than head stabilized
S l d h h d bl d
Body stabilized gave better understanding
Head tracking most intuitive

195. Field Studies

196. Field S d
F ld Studies
Field studies are done in natural settings
settings.
“in the wild” is a term for prototypes being used
freely in natural settings
settings.
Aim to understand what users do naturally and how
technology impacts them
them.
Field studies are used in product design to:
- identify opportunities for new technology;
- determine design requirements;
- decide how best to introduce new technology;gy;
- evaluate technology in use.
196 www.id-book.com

197. Observation
Direct observation in the field
Structuring frameworks
Degree of participation (i id or outsider)
D f ti i ti (insider t id )
Ethnography
Direct observation in controlled environments
D b ll d
Indirect observation: tracking users’ activities
Diaries
Interaction logging

198. Ethnography (1)
• Ethnography is a philosophy with a set of techniques that include
participant observation and interviews
• Debate about differences between participant observation and
ethnography
• Ethnographers immerse themselves in the culture that they study
• A researcher’s degree of participation can vary along a scale from
‘outside’ to ‘inside’
‘ d ’ ‘ d ’
• Analyzing video and data logs can be time-consuming
• Collections of comments, incidents, and artifacts are made

199. Ethnography (2)
Co operation
Co-operation of people being observed is required
Informants are useful
Data analysis is continuous
Interpretivist technique
Questions get refined as understanding grows
Reports usually contain examples

200. Direct observation in a controlled setting
g
Think-aloud technique
q
Indirect observation
Diaries
Interaction logs
Cultural
C lt l probesb

201. Frameworks to Guide Observation
- The person. Who?
- The place. Where?
- The thing What?
thing.
The Goetz and LeCompte (1984) framework:
- Who is present?
- What is their role?
- What is happening?
- When does the activity occur?
- Wh i it happening?
Where is h i ?
- Why is it happening?
- How is the activity organized?
201 www.id-book.com

202. Predictive Evaluation

203. Predictive models
Provide a way of evaluating products or
designs without directly involving users.
g y g
Less expensive than user testing.
Usefulness limited to systems with
predictable tasks - e.g., telephone answering
systems, mobiles,
systems mobiles cell phones etc
phones, etc.
Based on expert error-free behavior.

204. Fitts’ Law (Fitts 1954)
Fitts’ Law predicts that the time to point at an
object using a device is a function of the distance
from the target object and the object’s size.
The further away and the smaller the object, the
object
longer the time to locate it and point to it.

205. GOMS Model
Goals h
G l - the state the user wants to achieve e.g., find a
h hi fi d
website.
Operators - the cognitive processes and physical actions
needed to attain the goals
Eg moving mouse to select icon
g g
Methods - the procedures for accomplishing the goals, e.g.,
drag mouse over icon, click on button.
Selection rules - decide which method to select when there is
more than one.

206. GOMS Response Times (Card et al., 1983)
Operator Description Time (sec)
K Pressing a single key or button
g g y
Average skilled typist (55 wpm) 0.22
Average non-skilled typist (40 wpm) 0.28
Pressing shift or control key 0.08
Typist unfamiliar withthekeyboard
with the keyboard 1.20
120
P Pointing with a mouse or other device on a 0.40
display to select an object.
This value is derived fromFitts’ Law which is
discussed below.
P1 Clicking the mouse or similar device 0.20
H Bring ‘home’ hands on the keyboard or other 0.40
device
M Mentally prepare/respond 1.35
R(t) The response time is counted only if it causes t
the user to wait.

207. Expert Inspections
Several kinds.
Experts use their knowledge of users and
p g
technology to review software usability.
Expert critiques can be formal or informal reports.
p q p
Heuristic evaluation is a review guided by a set of
heuristics.
Walkthroughs involve stepping through a pre-
p
planned scenario noting potential problems.
gp p

208. Nielsen’s heuristics
Visibility of system status
status.
Match between system and real world.
User control and freedom
freedom.
Consistency and standards.
Error prevention.
E
Recognition rather than recall.
Flexibility and efficiency of use.
Aesthetic and minimalist design.g
Help users recognize, diagnose, recover from errors.
Help and documentation.

209. Three stages for doing heuristic evaluation
Briefing session to tell experts what to do.
Evaluation period of 1-2 hours in which:
Each expert works separately;
Take one pass to get a feel for the product;
Take a second pass to focus on specific features.
Debriefing session in which experts work
together to prioritize problems.
g p p

210. No. of evaluators & problems

211. Advantages and problems
Few ethical and practical issues to consider
because users not involved.
Can be difficult and expensive to find experts.
Best experts have knowledge of application
domain and users.
Biggest problems:
Important problems may get missed;
Many trivial problems are often identified;
Experts have biases.

212. Data Collection

213. Overview
• Fi k i
Five key issues of d gathering
f data h i
• Data recording
• Interviews
• Questionnaires
• Observation
• Choosing and combining techniques
213 www.id-book.com

214. Five K I
F Key Issues
1. Setting goals
- Decide how to analyze data once collected
2. Identifying participants
- Decide who to gather data from
3. Relationship with participants
- Clear and professional
- Informed consent when appropriate
pp p
4. Triangulation
- Look at data from more than one perspective
5. Pilot studies
- Small trial of main study
214 www.id-book.com

215. Data recording
Notes, audio, video, photographs
Notes plus photographs
Audio plus photographs
Video
215 www.id-book.com

216. Interviews
• Unstructured - are not directed by a script.
Rich but not replicable.
• St
Structured - are ti htl scripted, often like a
t d tightly i t d ft lik
questionnaire. Replicable but may lack
richness.
richness
• Semi-structured - guided by a script but
interesting issues can be explored in more
depth. Can provide a good balance between
richness and replicability.
216 www.id-book.com

217. Interview Q
i Questions
i
• Two types:
− ‘closed questions’ have a predetermined answer
format, e.g., ‘yes’ or ‘no’
− ‘open questions’ do not have a predetermined format
• Closed questions are easier to analyze
• A id
Avoid:
− Long questions
− Compound sentences - split them into two
− Jargon and language that the interviewee may not
understand
− Leading questions that make assumptions e.g., why do
you like …?
− Unconscious biases e.g., gender stereotypes
eg
217 www.id-book.com

218. Enriching the interview process
• Props - devices for prompting interviewee, e.g., a
prototype, scenario
218 www.id-book.com

219. Questionnaires
• Key Points
• Can be administered to large populations
• Paper email and the web used for dissemination
Paper,
• Sampling can be a problem when the size of a
population is unknown as is common online
• Design
• Provide clear instructions
• Questions can be closed or open
• closed easier to analyze
• The impact of a question can be influenced by
order
219 Will phrases be positive, negative or mixed
• www.id-book.com

220. Encouraging a Good Response
Make sure purpose of study is clear
y y
Promise anonymity
Ensure questionnaire is well designed
Offer a short version for those who do not have time to
complete a long questionnaire
Follow-up with emails, phone calls, letters
Provide an incentive
40% response rate is high, 20% is often acceptable
p g p
220 www.id-book.com

221. Online Questionnaires
• www.surveymonkey.com
k
• Good
• Low cost
• Responses are usually received quickly
• Data automatically collected
• Time required for data analysis is reduced
• Bad
• Sampling is problematic if population size is unknown
• Preventing individuals from responding more than once
221 www.id-book.com

222. 6. Design in the Real World
Mark Billinghurst

223. Product Development Balancing Act
Marketing g
Software
Hardware
Industrial Design
Interaction Design
Usability
Manufacturing

224. Obstacles to Success
Everyone has an opinion
E h i i
Design by committee
Misunderstanding user research
Too literal, not objective
Power struggles
Blue vs. Red Interface
Losing focus of the end user
Feature Creep
Relying on assumptions/myths
Featuritis = Requirements not well established

225. There is never only one design solution!!

226. Penny Wise = Dollar Poor
Software/Hardware development rewarded
g
for fast work and low budgets.
Interaction design activities take some budget
and some time
time.
Investment at front end saves money overall.

227. Cost Justifying Usability
Cost of Changes
Design Alternatives
Req. Dev. Build
Phase Phase Phase
Saving Development Costs

228. Cost Justifying Usability
Increase Revenue
“IBM’s Web presence has traditionally been made up of a difficult-to-navigate
labyrinth of disparate subsidies, but a redesign made it more cohesive and
subsidies
user-friendly. According to IBM, the massive redesign effort quickly paid
dividends. The company said in the month after re-launch that traffic to the
Shop IBM online store increased 120 percent and sales went up 400%400%.
(Battey, 1999)
Increase Safety
“Chapanis cites two independent studies that showed a 54% reduction
In rear-end accidents with the use of human factors improvement: the
centered high mount brake light on autos.”
high-mount autos.

229. More Value for Interaction Design
1. US trade magazines score products on usability along with
features and cost.
2. 1/3 of new products fail in market place. (PDMA)
3. Some big electronic chain stores do their own usability
review of products before they will stock them.
4. Customers have become very sophisticated and
DEMAND highly usable products (as do distributors, investors).
distributors investors)
5. Enables companies to develop on-going relationships with
customers to keep them current.
p

230. Stories from the Field
Connectix for Boeing
Assumption that users would
primarily be business travellers
wanting to work.
Studies showed primary user
were indeed business travellers
but wanting to RELAX.

231. In Conclusion…

232. 7. Resources

233. Books
Interaction Design by Jenny Preece, Yvonne
Rogers, and Helen Sharp
g p
About Face 2.0: The Essentials of Interaction
Design by Alan Cooper and Robert M. Reimann
M
Mobile Interaction Design by Matt Jones and
Gary Marsden

234. Resources: More books
Designing Interactions. Bill Moggridge
The Inmates Are Running the Asylum: Why High-Tech Products Drive
Us Crazy and How to Restore the Sanity. Alan Cooper
Interface Culture. Steven Johnson (history of the development of
interfaces, well written, more on the ideas)
The invisible Computer. D ld Norman
Th i i ibl C Donald N
Rapid Contextual Design. A How-to Guide to Key Techniques for User-
Centered Desi n Karen H lt blatt
Design. Holtzblatt
Design Research. Brenda Laurel

235. Resources: Conferences
OzCHI
CHI (Computer Human Interaction, ACM)
NordiCHI (Nordic CHI, ACM)
MobileHCI (focused on mobile interaction/devices, ACM)
IDC (Interaction Design and Children, ACM)
Interact
HCI International
DIS (Designing Interactive Systems, ACM) - Design Research, open to
different approaches t d i
diff t h to design
DPPI (Designing Pleasurable Products) - Product Design Conference
DUX (Designing the User Experience, ACM) - more practicioners
CSCW (Computer Supported Cooperative Work)
UbiComp

236. Online
http://www.interaction-design.org/ Interaction Design online
p g g g
encyclopedia
http://www.baddesigns.com/ scrapbook of illustrated examples
of things that are hard to use
http://www.ixda.org/en/ Interaction Design Association
http://dev.uxmatters.com/ (User Experience Matters - blog and
newsletter)
l )
http://www.nathan.com/ed/index.html (Nathan Shedroff‘s
website on Experience Design)
Web-Based User Interface Evaluation with Questionnaires, by
Gary Perlman http://www.acm.org/~perlman/question.html
http://usability.gov/ US D
h // bili / Dept. Of Health and H
H l h d Human Service
S i
http://www.paperprototyping.com/ (adjunct homepage to book
on paper p
p p prototyping)
yp g)

237. Institutions
D-School - Stanford University (USA)
http://www.stanford.edu/group/dschool/
IIT Institute of Design (USA)
http://www.id.iit.edu/
CMU School of Design (USA)
www.design.cmu.edu/
MIT Media Lab (USA)
www.media.mit.edu
Interaction Design Institute + Domus Academy (Italy)
I i D i I i D A d (I l )
http://www.interaction-ivrea.it/
Interaction Design, Royal Co ege of Art (UK)
te act o es g , oya College o t (U )
http://www.interaction.rca.ac.uk/
University of Queensland (Australia)
http://www.itee.uq.edu.au/~id/
htt // it d / id/

238. More Information
• Mark Billinghurst
– mark billinghurst@hitlabnz org
mark.billinghurst@hitlabnz.org
• Website
– www.hitlabnz.org