This was a course taught at the CHI 2014 conference on May 1st by Mark Billinghurst (HIT Lab NZ) and Hayes Raffle (Google). It teaches the fundamentals of designing wearable interfaces.
Generative AI for Technical Writer or Information Developers
The Glass Class: Designing Wearable Interfaces
1. The Glass Class
Designing Wearable Interfaces
May 1, CHI 2014
Mark Billinghurst
HIT Lab NZ
University of Canterbury
mark.billinghurst@canterbury.ac.nz
Hayes Raffle
Glass team
Google [x]
hraffle@google.com
3. Mark Billinghurst
▪ Director of HIT Lab NZ, University
of Canterbury
▪ PhD Univ. Washington
▪ Research on AR, mobile HCI,
Collaborative Interfaces
▪ More than 250 papers in AR, VR,
interface design
▪ Sabbatical in Glass team at
Google [x] in 2013
4. Hayes Raffle
▪ Interaction Research Lead,
Google Glass
▪ PhD MIT Media Lab
▪ Ran a couple of companies
▪ Launched a few products
▪ Won a few awards
▪ Published many papers in HCI
9. Course Goals
In this course you will learn
▪ Introduction to head mounted wearable computers
▪ Understanding of current wearable technology
▪ Key design principles/interface metaphors
▪ Relevant human perceptual principles
▪ Rapid prototyping tools
▪ Areas for future research
▪ Hands on experience with the technology
10. What You Won’t Learn
▪ Low level programming
▪ Glass Mirror API, GDK, Vuzix SDK, etc
▪ Designing for non-HMD based interfaces
▪ Watches, fitness bands, etc
▪ How to develop wearable hardware
▪ optics, sensor assembly, etc
▪ Evaluation methods
▪ Experimental design, statistics, etc
11. Schedule
9:00 Introduction (Mark + Hayes)
9:05 Overview/History (Mark)
9:20 Evolution and Design Principles (Hayes)
9:45 Prototyping Tools + Best Practices (Mark / Hayes)
10:20 Break/Demo
10:30 Concept Design Exercise (Design group / Hayes)
10:50 Wearable Technologies (Lecture group / Mark)
11:20 Design Presentations (Design group / Hayes)
11:50 Research Directions (Mark + Hayes)
12:20 Finish
12. Design Group (33 People)
If your name is on this list you are in the Design Group
Yang Wang
Konstantino Kapetaneas
Preethi Srinivas
Tony James
Kate Vogt
Aneesh Tarun
Josh Andres
Maria Maimó
Bram Reurings
Luke Mill
Tuck-Voon How
M Gill
Janaki Kumar
Melinda Knight
M Calkins
Mike Tissenbaum
Samantha Tse
Kal McDowd
Adora Tam
Oscar Meruvia
Mike Chen
Anita Hoechtl
Merlin Stone
Ashoomi Dohlakia
Icy Zhu
Zdenek Mikovec
Cristina Manresa-Yee
Christian Winkler
Angela Noh
Amyris Fernandez
Deborah Ptak
Arne Renkema-Padmos
Thomas Fritz
13. Display Demos You Can Try
Google Glass Display
Glass UI, AR demos, Games, multimedia capture
Vuzix M-100 Display
Monocular display
Epson BT-100, Epson BT-200
See through displays, Junaio markerless tracking
Brother AirScouter display
Projected see-through image
Recon Snow
Micro-display integrated into ski goggles
16. A Brief History of Time
▪ Trend
▪ smaller, cheaper, more functions, more intimate
▪ Time pieces moved from public space onto the body
18th
Century
20th
Century
13th
Century
17. A Brief History of Computing
Trend
▪ Smaller, cheaper, faster, more intimate
▪ Moving from fixed to handheld and onto body
1950’s
1980’s
1990’s
19. What is a Wearable Computer ?
▪ A computer that is:
▪ Portable while operational
▪ Enables hands-free/hands-limited use
▪ Able to get the user’s attention
▪ Is always on, acting on behalf of the user
▪ Able to sense the user’s current context
Rhodes, B. J. (1997). The wearable remembrance agent: A system for
augmented memory. Personal Technologies, 1(4), 218-224.
20. In Other Words ..
▪ A computer that is ..
▪ Eudaemonic: User considers it part of him/herself
▪ Existential: User has complete control of the system
▪ Ephemeral: System always operating at some level
Mann, S. (1997). Wearable computing: A first step toward personal
imaging. Computer, 30(2), 25-32.
21. Wearable Computing
▪ Computer on the body that is:
▪ Always on
▪ Always accessible
▪ Always connected
▪ Other attributes
▪ Augmenting user actions
▪ Aware of user and surroundings
22. Augmented Interaction
Rekimoto, J., & Nagao, K. (1995, December). The world through the computer:
Computer augmented interaction with real world environments. In Proceedings of the
8th annual ACM symposium on User interface and software technology (pp. 29-36).
23. The Ideal Wearable
▪ Persists and Provides Constant Access: Designed for
everyday and continuous user over a lifetime.
▪ Senses and Models Context: Observes and models
the users environment, mental state, it’s own state.
▪ Augments and Mediates: Information support for the
user in both the physical and virtual realities.
▪ Interacts Seamlessly: Adapts its input and output
modalities to those most appropriate at the time.
Starner, T. E. (1999). Wearable computing and contextual awareness
(Doctoral dissertation, Massachusetts Institute of Technology).
25. History of Wearables
▪ 1960-90: Early Exploration
▪ Custom build devices
▪ 1990 - 2000: Academic, Military Research
▪ MIT, CMU, Georgia Tech, EPFL, etc
▪ 1997: ISWC conference starts
▪ 1995 – 2005+: First Commercial Uses
▪ Niche industry applications, Military
▪ 2010 - : Second Wave of Wearables
▪ Consumer applications, Head Worn
26. Thorp and Shannon (1961)
▪ Wearable timing device for roulette prediction
▪ Audio feedback, four button input
Ed Thorp
Thorp, E. O. (1998, October). The invention of the first wearable computer. In
Wearable Computers, 1998. Second International Symposium on (pp. 4-8). IEEE.
27. Keith Taft (1972)
▪ Wearable computer for blackjack card counting
▪ Toe input, LED in Glasses for feedback
Belt computer Shoe Input Glasses Display
34. Early Technology
▪ Computing
▪ Belt or Backpack
▪ Displays
▪ Head Mounted, LCD Panel, Audio
▪ Input Devices
▪ Chording Keyboard, Speech, Camera
▪ Networking
▪ Wireless LAN, Infra-Red, Cellular
35. US Military Wearables (1989- )
▪ Early experimentation
▪ 386 computer, VGA display
▪ GPS, mapping software
▪ Land Warrior (1991-)
▪ Integrated wearable system
▪ Camera, colour display, radio
▪ Navigation, reports, photos
Zieniewicz, M. J., Johnson, D. C., Wong, C., & Flatt, J. D. (2002). The evolution of
army wearable computers. IEEE Pervasive Computing, 1(4), 30-40.
36. Wearables at CMU (1991–2000)
▪ Industry focused wearables
▪ Maintenance, repair
▪ Custom designed interface
▪ Dial/button input
▪ Rapid prototyping approach
▪ Industrial designed, ergonomic
http://www.cs.cmu.edu/afs/cs/project/vuman/www/frontpage.html
37.
38. Early Commercial Systems
▪ Xybernaut (1996 - 2007)
▪ Belt worn, HMD, 200 MHz
▪ ViA (1996 – 2001)
▪ Belt worn, Audio Interface
▪ 700 MHz Crusoe
■ Symbol (1998 – 2006)
■ Wrist worn computer
■ Finger scanner
40. Mobile AR: Touring Machine (1997)
▪ University of Columbia
▪ Feiner, MacIntyre, Höllerer, Webster
▪ Combines
▪ See through head mounted display
▪ GPS tracking
▪ Orientation sensor
▪ Backpack PC (custom)
▪ Tablet input
Feiner, S., MacIntyre, B., Höllerer, T., & Webster, A. (1997). A touring machine: Prototyping 3D mobile
augmented reality systems for exploring the urban environment. Personal Technologies, 1(4), 208-217.
41. MARS View
▪ Virtual tags overlaid on the real world
▪ “Information in place”
42. Backpack/Wearable Systems
1997 Backpack Wearables
▪ Feiner’s Touring Machine
▪ AR Quake (Thomas)
▪ Tinmith (Piekarski)
▪ MCAR (Reitmayr)
▪ Bulky, HMD based
Piekarski, W., & Thomas, B. (2002). ARQuake: the outdoor
augmented reality gaming system. Communications of the
ACM, 45(1), 36-38.
43. PCI 3D Graphics Board
Hard Drive
Serial
Ports
CPU
PC104 Sound Card
PC104 PCMCIA
GPS
Antenna
RTK correction Antenna
HMD
Controller
Tracker
Controller
DC to DC
Converter
Battery
Wearable
Computer
GPS RTK
correction
Radio
Example self-built working
solution with PCI-based 3D graphics
Columbia Touring Machine
Mobile AR - Hardware
44. HIT Lab NZ Wearable AR (2004)
▪ Highly accurate outdoor AR
tracking system
▪ GPS, Inertial, RTK system
▪ HMD
▪ First prototype
▪ Laptop based
▪ Video see-through HMD
▪ 2-3 cm tracking accuracy
46. 2009 - Layar (www.layar.com)
• Location based data
– GPS + compass location
– Map + camera view
• AR Layers on real world
– Customized data
– Audio, 3D, 2D content
• Easy authoring
• Android, iPhone
47. Wearable Evolution
Backpack+HMD:
…10+ kg
Handheld + HMD
… Separate sensors
.... UMPC 1.1GHz
…1.5kg
…still >$5K
Scale it down more:
Smartphone…$500
…Integrated
…0.1kg
…billions of units
1997 2003 2007
57. Summary
Wearables are a new class of computing
Intimate, persistent, aware, accessible, connected
Evolution over 50 year history
Backpack to head worn
Custom developed to consumer ready device
Enables new applications
Collaboration, memory, AR, industry, etc
Many head worn wearables are coming
Android based, sensor package, micro-display
64. "Computing
should just be
more comfortable"
"Google should do the hard
work, and you should have a
chance to live, have a good
life, and get on with it."
65. As technology becomes more
personal and immediate, it can
start to disappear.
Distant Intimate
83. Transparent displays are tricky
Colors are funny and inconsistent.
You can only add light to a scene, not cover anything up.
Motion can be disorienting.
Clarity, contrast, brightness, visual field and attention are important.
89. The world is the experience
Get the interface and interactions out of the way.
90.
91. Micro
Interactions
The position of the display and
limited input ability makes
longer interactions less
comfortable.
Using it shouldn’t take longer
than taking out your phone.
92.
93. A rear view mirror
Don't overload the user. Stick to the
absolutely essential, avoid long
interactions. Be explicit.
94.
95. As personal as
it gets
Recognize and adapt to the
user… not the other way
around.
Glass is the most personal device you own. It
operates closer to your most valuable senses
and your environment, and it knows more about
both.
Glass should adapt, be personal and recognize
the wearer; be aware of what consists his
identity (physiology, memory, preferences,
environment) and connect to it.
As personal as it gets
Glass is the most personal device you own.
It should recognize and adapt to you…
not the other way around.
96. For the closest people
and most important
moments
Glass should be for prioritizing your
closest people and creating value
for the whole group, not just the
wearer.
104. A world of stories
In Viewpoint of Billions by David
Datuna, Glass allows viewers to
unlock images and video with
interactive experiences.
105. Social action
First-person journalist Tim Pool
broadcasts an intimate view
of Istanbul protests.
'I want to show you what it's like to be there as best I can, even
if that ends with me running full-speed into a cafe and rubbing
lemons all over my face after being tear-gassed'
114. Important Note
▪ Most current wearables run Android OS
▪ eg Glass, Vuzix, Atheer, Epson, etc
▪ So many tools for prototyping on Android
mobile devices will work for wearables
▪ If you want to learn to code, learn
▪ Java, Android, Javascript/PHP
115. Typical Development Steps
▪ Sketching
▪ Storyboards
▪ UI Mockups
▪ Interaction Flows
▪ Video Prototypes
▪ Interactive Prototypes
▪ Final Native Application
Increased
Fidelity &
Interactivity
125. Limitations
▪ Positives
▪ Good for documenting screens
▪ Can show application flow
▪ Negatives
▪ No interactivity/transitions
▪ Can’t be used for testing
▪ Can’t deploy on wearable
▪ Can be time consuming to create
128. ▪Series of still photos in a movie format.
▪Demonstrates the experience of the product
▪Discover where concept needs fleshing out.
▪Communicate experience and interface
▪You can use whatever tools, from Flash to iMovie.
Video Sketching
133. UXpin - www.uxpin.com
▪ Web based wireframing tool
▪ Mobile/Desktop applications
▪ Glass templates, run in browser
https://www.youtube.com/watch?v=0XtS5YP8HcM
134. Proto.io - http://www.proto.io/
▪ Web based mobile prototyping tool
▪ Features
▪ Prototype for multiple devices
▪ Gesture input, touch events, animations
▪ Share with collaborators
▪ Test on device
140. Justinmind - http://www.justinmind.com/
▪ Native wireframing tool
▪ Build mobile apps without programming
▪ drag and drop, interface templates
▪ web based simulation
▪ test on mobile devices
▪ collaborative project sharing
▪ Templates for Glass, custom templates
143. Comparing Wireframe Tools
Tool Web Native Wearable
Template
Interaction
Uxpin X X
Proto.io X X
Justinmind X X X
Axure X X X
144. Wireframe Limitations
▪ Can’t deploy on Glass
▪ No access to sensor data
▪ Camera, orientation sensor
▪ No multimedia playback
▪ Audio, video
▪ Simple transitions
▪ No conditional logic
▪ No networking
146. Processing
▪ Programming tool for Artists/Designers
▪ http://processing.org
▪ Easy to code, Free, Open source, Java based
▪ 2D, 3D, audio/video support
▪ Processing For Android
▪ http://wiki.processing.org/w/Android
▪ Strong Android support
▪ Generates Android .apk file
147. Processing - Motivation
▪ Language of Interaction
▪ Sketching with code
▪ Support for rich interaction
▪ Large developer community
▪ Active help forums
▪ Dozens of plug-in libraries
▪ Strong Android support
▪ Easy to run on wearables
151. Basic Parts of a Processing Sketch
/* Notes comment */
//set up global variables
float moveX = 50;
//Initialize the Sketch
void setup (){
}
//draw every frame
void draw(){
}
152. Importing Libraries
▪ Can add functionality by Importing Libraries
▪ java archives - .jar files
▪ Include import code
import processing.opengl.*;
▪ Popular Libraries
▪ Minim - audio library
▪ OCD - 3D camera views
▪ Physics - physics engine
▪ bluetoothDesktop - bluetooth networking
154. Processing and Glass
▪ One of the easiest ways to build rich
interactive wearable applications
▪ focus on interactivity, not coding
▪ Collects all sensor input
▪ camera, accelerometer, touch
▪ Can build native Android .apk files
▪ Side load onto Glass
155. Example: Hello World
//called initially at the start of the Processing sketch
void setup() {
size(640, 360);
background(0);
}
//called every frame to draw output
void draw() {
background(0);
//draw a white text string showing Hello World
fill(255);
text("Hello World", 50, 50);
}
157. Hello World Image
PImage img; // Create an image variable
void setup() {
size(640, 360);
//load the ok glass home screen image
img = loadImage("okGlass.jpg"); // Load the image into
the program
}
void draw() {
// Displays the image at its actual size at point (0,0)
image(img, 0, 0);
}
159. Touch Pad Input
▪ Tap recognized as DPAD input
void keyPressed() {
if (key == CODED){
if (keyCode == DPAD) {
// Do something ..
▪ Java code to capture rich motion events
▪ import android.view.MotionEvent;
160. Motion Event
//Glass Touch Events - reads from touch pad
public boolean dispatchGenericMotionEvent(MotionEvent event) {
float x = event.getX(); // get x/y coords
float y = event.getY();
int action = event.getActionMasked(); // get code for action
switch (action) { // let us know which action code shows up
case MotionEvent.ACTION_DOWN:
touchEvent = "DOWN";
fingerTouch = 1;
break;
case MotionEvent.ACTION_MOVE:
touchEvent = "MOVE";
xpos = myScreenWidth-x*touchPadScaleX;
ypos = y*touchPadScaleY;
break;
162. Sensors
▪ Ketai Library for Processing
▪ https://code.google.com/p/ketai/
▪ Support all phone sensors
▪ GPS, Compass, Light, Camera, etc
▪ Include Ketai Library
▪ import ketai.sensors.*;
▪ KetaiSensor sensor;
163. Using Sensors
▪ Setup in Setup( ) function
▪ sensor = new KetaiSensor(this);
▪ sensor.start();
▪ sensor.list();
▪ Event based sensor reading
void onAccelerometerEvent(…)
{
accelerometer.set(x, y, z);
}
165. Using the Camera
▪ Import camera library
▪ import ketai.camera.*;
▪ KetaiCamera cam;
▪ Setup in Setup( ) function
▪ cam = new KetaiCamera(this, 640, 480, 15);
▪ Draw camera image
void draw() {
//draw the camera image
image(cam, width/2, height/2);
}
167. Timeline Demo
▪ Create Card Class
▪ load image, card number, children/parent cards
▪ Timeline Demo
▪ Load cards in order
▪ Translate cards with finger motion
▪ Swipe cards in both directions
▪ Snap cards into position
169. Overview
▪ For best performance need native coding
▪ Low level algorithms etc
▪ Most current wearables based on Android OS
▪ Need Java/Android skills
▪ Many devices have custom API/SDK
▪ Vusix M-100: Vusix SDK
▪ Glass: Mirror API, Glass Developer Kit (GDK)
172. Glassware and Timeline
▪ Static Cards
▪ Static content with text, HTML, images, and video.
▪ e.g. notification messages, news clip
▪ Live Cards
▪ Dynamic content updated frequently.
▪ e.g. compass, timer
▪ Immersions
▪ Takes over the whole control, out from timeline.
▪ e.g. interactive game
173. Glassware Development
▪ Mirror API
▪ Server programming, online/web application
▪ Static cards / timeline management
▪ GDK
▪ Android programming, Java (+ C/C++)
▪ Live cards & Immersions
▪ See: https://developers.google.com/glass/
174. ▪ REST API
▪ Java servlet, PHP, Go,
Python, Ruby, .NET
▪ Timeline based apps
▪ Static cards
- Text, HTML, media attachment (image & video)
- Standard and custom menu items
▪ Manage timeline
- Subscribe to timeline notifications
- Sharing with contacts
- Location based services
Mirror API
175. GDK
▪ Glass Development Kit
▪ Android 4.0.3 ICS + Glass specific APIs
▪ Use standard Android Development Tools
176. ▪ GDK add-on features
▪ Timeline and cards
▪ Menu and UI
▪ Touch pad and gesture
▪ Media (sound, camera and voice input)
GDK
177. Glass Summary
▪ Use Mirror API if you need ...
▪ Use GDK if you need ...
▪ Or use both
182. Rasberry Pi Glasses
▪ Modify video glasses, connect to Rasberry Pi
▪ $200 - $300 in parts, simple assembly
▪ https://learn.adafruit.com/diy-wearable-pi-near-eye-kopin-video-
glasses
183. Physical Input Devices
▪ Can we develop unobtrusive input devices ?
▪ Reduce need for speech, touch pad input
▪ Socially more acceptable
▪ Examples
▪ Ring,
▪ pendant,
▪ bracelet,
▪ gloves, etc
188. Summary
▪ Prototyping for wearables is similar to mobiles
▪ Tools for UI design, storyboarding, wireframing
▪ Android tools to create interactive prototypes
▪ App Inventor, Processing, etc
▪ Arduino can be used for hardware prototypes
▪ Once prototyped Native Apps can be built
▪ Android + SDK for each platform
189. Other Tools
▪ Wireframing
▪ pidoco
▪ FluidUI
▪ Rapid Development
▪ Phone Gap
▪ AppMachine
▪ Interactive
▪ App Inventor
▪ WearScript
190. App Inventor - http://appinventor.mit.edu/
▪ Visual Programming for Android Apps
▪ Features
▪ Access to Android Sensors
▪ Multimedia output
▪ Drag and drop web based interface
▪ Designer view – app layout
▪ Blocks view – program logic/control
196. ✓ Don’t design an app
Glass OS is time-based model, not an app model.
X
197. ✓ Know what makes Glass different than a phone
Glass has certain superpowers. Remember what these
superpowers are and use them to augment the experience you’re
designing.
198. X
Don’t just port your mobile
experience over to Glass
It won’t work. It will be too busy. It will be hard for users to
quickly understand and navigate your content. The
constraints on Glass are simply too strict.
Bad idea. Even inverted, aside from
the need to properly format each
screen, the layout simply contains
too much information for Glass.
Good idea!
199. ✓ Design for the (hyper)now
When is
my next
meeting?
How many
calories have I
eaten today? Can I
get a burger for
lunch?
Spend 90% of your time thinking about what people want to know
(in sport or elsewhere) at any given moment. The more you know
about what info people need and currently don’t have - the more
compelling your design will be.
201. ✓ Design for emotion
thumbs up viewsnap - running
202. ✓ Make it glanceable
Seek to rigorously reduce information density. Successful designs
afford for recognition, not reading.
Bad Good
203. ✓ Reduce the number of info chunksX✓
You are designing for recognition, not reading. Reducing the total # of
information chunks will greatly increase the glanceability of your design.
1
2
3
1
2
3
4
5 (6)
Test done by Morten Just using a watch
204. ✓ Design single interactions to be faster than 4 s
Eye movements
For 1: 1 230ms
For 2: 1 230ms
For 3: 1 230ms
For 4: 3 (52/17) 690ms
For 5: 2 460ms
~1,840ms
Eye movements
For 1: 1-2 460ms
For 2: 1 230ms
For 3: 1 230ms
~920ms
1
2
3
1
2
3
4
5 (6)
Test done by Morten Just using a watch
208. ✓ On the device, black is not blackX
Your mock Device simulation
209. ✓ Establish hierarchy with color - not font size
White is your <h1> and grey is your <h2> or <h3>. Footer text -
establishing time, attribution, or distance - is the only place where
you see a smaller font size used.
210. ✓ If you have brand-specific typography - use it
211. ✓ Use relative information display
calendar card - in 10 minutes, Fri - viewsnap
212. ✓
Remember, people have an ever-growing
ecosystem of wearablesX
Each device should be used when it’s most relevant and when it’s
the easiest interaction available.
213. ✓ The Glass screen is just one part of the
experience
214. ✓ Do view your design on the device
1. Download Android Design Preview
2. Plug in Glass
3. Terminal command: java -jar AndroidDesignPreview-0.3.2.jar
4. Drag the red rectangle over your work to take a look
5. Check out the developer site for more in-depth guidelines
216. Design Group (33 People)
▪ If your name is on this list you are in the Design Group
Yang Wang
Konstantino Kapetaneas
Preethi Srinivas
Tony James
Kate Vogt
Aneesh Tarun
Josh Andres
Maria Maimó
Bram Reurings
Luke Mill
Tuck-Voon How
M Gill
Janaki Kumar
Melinda Knight
M Calkins
Mike Tissenbaum
Samantha Tse
Kal McDowd
Adora Tam
Oscar Meruvia
Mike Chen
Anita Hoechtl
Merlin Stone
Ashoomi Dohlakia
Icy Zhu
Zdenek Mikovec
Cristina Manresa-Yee
Christian Winkler
Angela Noh
Amyris Fernandez
Deborah Ptak
Arne Renkema-Padmos
Thomas Fritz
221. Key Properties of HMD
▪ Field of View
▪ Human eye 95 deg. H, 60/70 deg. V
▪ Resolution
▪ > 320x240 pixel
▪ Refresh Rate
▪ Focus
▪ Fixed/manual
▪ Size, Weight
▪ < 350g for long term
▪ Power
222. Types of Head Mounted Displays
Occluded
See-thru
Multiplexed
226. Strengths of optical see-through
▪ Simpler (cheaper)
▪ Direct view of real world
▪ Full resolution, no time delay (for real world)
▪ Safety
▪ Lower distortion
▪ No eye displacement
▪ see directly through display
229. Vuzix Wrap 1200DXAR
▪ Stereo video see-through display ($1500)
▪ Twin 852 x 480 LCD displays, 35 deg. FOV
▪ Stereo VGA cameras
▪ 3 DOF head tracking
230. Strengths of Video See-Through
▪ True occlusion
▪ Block image of real world
▪ Digitized image of real world
▪ Flexibility in composition
▪ Matchable time delays
▪ More registration, calibration strategies
▪ Wide FOV is easier to support
▪ wide FOV camera
231. Multiplexed Displays
▪ Above or below line of sight
▪ Strengths
▪ User has unobstructed view of real world
▪ Simple optics/cheap
▪ Weaknesses
▪ Direct information overlay difficult
▪ Display/camera offset from eyeline
▪ Wide FOV difficult
232. Vuzix M-100
▪ Monocular multiplexed display ($1000)
▪ 852 x 480 LCD display, 15 deg. FOV
▪ 5 MP camera, HD video
▪ GPS, gyro, accelerometer
233. Display Types
▪ Curved Mirror
▪ off-axis projection
▪ curved mirrors in front of eye
▪ high distortion, small eye-box
▪ Waveguide
▪ use internal reflection
▪ unobstructed view of world
▪ large eye-box
234. See-through thin displays
▪ Waveguide techniques for thin see-through displays
▪ Wider FOV, enable AR applications
▪ Social acceptability
Opinvent Ora
240. Twiddler Input
▪ Chording or multi-tap input
▪ Possible to achieve 40 - 60 wpm after 30+ hours
▪ Chording input about 50% faster than multi-tap
▪ cf 20 wpm on T9, or 60+ wpm for QWERTY
Lyons, K., Starner, T., Plaisted, D., Fusia, J., Lyons, A., Drew, A., & Looney, E. W. (2004, April).
Twiddler typing: One-handed chording text entry for mobile phones. In Proceedings of the SIGCHI
conference on Human factors in computing systems (pp. 671-678). ACM.
241. Virtual Keyboards
▪ In air text input
▪ Virtual QWERTY keyboard up to 20 wpm
▪ On real keyboard around 45-60+ wpm
▪ Word Gesture up to 28 wpm
▪ On tablet/phone Word Gesture up to 47 wpm
▪ Handwriting around 20-30 wpm
A. Markussen, et. al. Vulture: A Mid-Air Word-Gesture Keyboard (CHI 2014)
242. Unobtrusive Input Devices
▪ GestureWrist
▪ Capacitive sensing
▪ Change signal depending on hand shape
Rekimoto, J. (2001). Gesturewrist and gesturepad: Unobtrusive wearable interaction devices. In
Wearable Computers, 2001. Proceedings. Fifth International Symposium on (pp. 21-27). IEEE.
244. Skinput
Using EMG to detect muscle activity
Tan, D., Morris, D., & Saponas, T. S. (2010). Interfaces on the go. XRDS:
Crossroads, The ACM Magazine for Students, 16(4), 30-34.
245. Issues to Consider
▪ Fatigue
▪ “Gorrilla” Arm from free-hand input
▪ Comfort
▪ People want to do small gestures by waist
▪ Interaction on the go
▪ Can input be done while moving?
246. Interaction on the Go
▪ Fitt’s law still applies while interacting on the go
▪ Eg: Tapping while walking reduces speed by > 35%
▪ Increased errors while walking
Lin, M., Goldman, R., Price, K. J., Sears, A., & Jacko, J. (2007). How do people tap when walking? An
empirical investigation of nomadic data entry.International Journal of Human-Computer Studies, 65(9),
759-769.
249. Spatial Cues for Wearable Info
Billinghurst, M., Bowskill, J., Dyer, N., & Morphett, J. (1998, March). An evaluation of wearable
information spaces. In Virtual Reality Annual International Symposium, 1998. Proceedings., IEEE
1998 (pp. 20-27). IEEE.
▪ Spatial cues sign. improve performance
▪ No difference between audio and visual cues
250. Spatial Conferencing
3+ attendees can be distinguished with spatialized audio
but could not without spatialized audio
Billinghurst, M., Bowskill, J., Jessop, M., & Morphett, J. (1998, October). A wearable spatial
conferencing space. In Proceedings of ISWC, 1998. (pp. 76-83). IEEE.
251. Organizing Tools in a Halo Display
Biocca, F., Tang, A., Lamas, D., Gregg, J., Brady, R., & Gai, P. (2001). How do users organize virtual
tools around their body in immersive virtual and augmented environment?: An exploratory study of
egocentric spatial mapping of virtual tools in the mobile infosphere. Media Interface and Network Design
Labs, Michigan State University, East Lansing, MI.
253. Notification Interruptions
▪ Gradually increase engagement
▪ Reduce attention load
Receiving SMS on Glass
“Bing”
Tap
Swipe
Glass
Show Message Start Reply
User
Look
Up
Say
Reply
254. Nomadic Radio (2000)
▪ Spatial audio wearable interface
Sawhney, N., & Schmandt, C. (2000). Nomadic radio: speech and audio interaction for contextual
messaging in nomadic environments. ACM transactions on Computer-Human interaction (TOCHI), 7
(3), 353-383.
259. AR Perceptual Issues
❖ Environment: Issues related to the environment itself.
❖ Capturing: Issues related to digitizing the environment
❖ Augmentation: Issues related to the design, layout, and
registration or AR content
❖ Display device: Technical issues associated with the display
device.
❖ User: Issues associated with user perceiving content.
E. Kruijff, J. E. Swan, and S. Feiner. Perceptual issues in augmented reality revisited.
9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2010, pp. 3--12.
260. Cognitive Issues in Mobile AR
▪ Three categories of issues
▪ Information Presentation – displaying virtual
information on the real world
▪ Physical Interaction – content creation,
manipulation and navigation
▪ Shared Experience – collaboration and
supporting common experiences
Li, Nai, and Henry Been-Lirn Duh. "Cognitive Issues in Mobile Augmented
Reality: An Embodied Perspective." Human Factors in Augmented Reality
Environments. Springer New York, 2013. 109-135.
262. Use the Following Depth Cues
▪ Movement parallax.
▪ Icon/Object size (for close objects)
▪ Linear perspective
▪ To add side perspective bar.
▪ Overlapping
▪ Works if the objects are big enough
▪ Shades and shadows.
▪ Depends on the available computation
263. Information Presentation
• Amount of information
• Clutter, complexity
• Representation of information
• Navigation cues, POI representation
• Placement of information
• Head, body, world stabilized
• View combination
• Multiple views
268. Text Representation
▪ Object space
▪ In general, does not fit wearable applications
▪ Billboards
▪ Screen space
▪ Annotations placed on a 2D plane, usually
parallel to the projection plane.
▪ Better legibility of text.
269. Outdoor background textures
Changing outdoor illuminanation, and text drawing styles.
Billboard and green text drawing styles are recommended.
Active text drawing styles did not perform better relative to
static styles.
272. Multi-View Navigation
▪ Use multiple views (ego vs exo-centric)
▪ Map/top down view – long distance navigation
▪ AR View – nearby navigation, situational awareness
▪ Radar view in AR view
276. Resource Competition Framework
▪ Mobility tasks compete for cognitive resources
with other tasks
▪ the most important given higher priority
▪ RCF is a method for analyzing this, based on:
▪ task analysis
▪ modelling cognitive resources
▪ a resource approach to attention
Oulasvirta, A., Tamminen, S., Roto, V., & Kuorelahti, J. (2005, April). Interaction in 4-second bursts: the
fragmented nature of attentional resources in mobile HCI. In Proceedings of the SIGCHI conference on
Human factors in computing systems (pp. 919-928). ACM.
277. RCF Key Assumptions
Four Key Assumptions
1. Functional Modularity: cognitive system divided into
functionally separate systems with diff. representations
2. Parallel Module Operation: cognitive modules operate
in parallel, independent of each other
3. Limited Capacity: cognitive modules are limited in
capacity with respect to time or content
4. Serial Central Operation: central coordination of
modules (eg monitoring) is serial
278. Cognitive Interference
▪ Structural interference
▪ Two or more tasks compete for limited
resources of a peripheral system
▪ eg two cognitive processes needing vision
▪ Capacity interference
▪ Total available central processing
overwhelmed by multiple concurrent tasks
▪ eg trying to add and count at same time
280. Using RCF
1. Map cognitive faculty to task
2. Look for conflicts/overloads
3. Analyse for competition for attention
4. Look for opportunities for technology to
reduce conflicts/competition
283. Application of RCF
Busy street > Escalator > Café > Laboratory.
But if you made Wayfinding, Path Planning, Estimating Time
to Target, Collision Avoidance easier?
285. Where to put Wearables?
▪ Places for unobtrusive wearable technology
Gemperle, F., Kasabach, C., Stivoric, J., Bauer, M., & Martin, R. (1998, October). Design for wearability. In
Wearable Computers, 1998. Digest of Papers. Second International Symposium on (pp. 116-122). IEEE.
286. Where to Place Trackpad?
▪ User study 25 people different postures
▪ Front of thigh most preferred, torso/upper arm worst
Thomas, Bruce, et al. "Determination of placement of a body-attached mouse as a pointing input
device for wearable computers." 2012 16th International Symposium on Wearable Computers. IEEE
Computer Society, 1999.
288. Tool/Task Design Paradigm
Lin, R., & Kreifeldt, J. G. (2001). Ergonomics in wearable computer design.
International Journal of Industrial Ergonomics, 27(4), 259-269.
320. Challenges for the Future (2001)
▪ Privacy
▪ Power use
▪ Networking
▪ Collaboration
▪ Heat dissipation
▪ Interface design
▪ Intellectual tools
▪ Augmented Reality systems
Starner, T. (2001). The challenges of wearable computing: Part 1. IEEE Micro,21(4), 44-52.
Starner, T. (2001). The challenges of wearable computing: Part 2. IEEE Micro,21(4), 54-67.
323. Capturing Behaviours
▪ 3 Gear Systems
▪ Kinect/Primesense Sensor
▪ Two hand tracking
▪ http://www.threegear.com
324. Gesture Interaction With Glass
▪ 3 Gear Systems
▪ Hand tracking
▪ Hand data sent to glass
▪ Wifi networking
▪ Hand joint position
▪ AR application rendering
▪ Vuforia tracking
325. Performance
▪ Full 3d hand model input
▪ 10 - 15 fps tracking, 1 cm fingertip resolution
326. User Study
▪ Gesture vs. Touch pad vs. Combined input
▪ Gesture 3x faster, no difference in accuracy
338. Context Sensing
▪ Using context to manage information
▪ progressive information display as user shows
interest
▪ Context from
▪ Speech
▪ Gaze
▪ Real world
▪ Wearable AR Display
Ajanki, A., Billinghurst, M., Gamper, H., Järvenpää, T., Kandemir, M., Kaski, S., ... & Tossavainen, T.
(2011). An augmented reality interface to contextual information. Virtual reality, 15(2-3), 161-173.