This document provides an overview of augmented reality technology. It defines augmented reality as combining real and virtual images in real-time and in 3D. Various AR display technologies are described, including head-mounted displays, spatial displays that project onto surfaces, and handheld/mobile displays. Key considerations for different display types like field of view and resolution are also outlined. The document discusses the core technologies needed for AR like tracking and interactive input. A variety of applications of AR are mentioned along with models for AR experience design.

1. COSC 426: Augmented Reality
Mark Billinghurst
mark.billinghurst@hitlabnz.org
July 19th 2013
Lecture 2: AR Technology

2. Key Points from Lecture 1

3. Augmented Reality Definition
 Defining Characteristics [Azuma 97]
 Combines Real and Virtual Images
- Both can be seen at the same time
 Interactive in real-time
- Virtual content can be interacted with
 Registered in 3D
- Virtual objects appear fixed in space

4. What is not Augmented Reality?
 Location-based services
 Barcode detection (QR-codes)
 Augmenting still images
 Special effects in movies
 …
 … but they can be combined with AR!

5. Milgram’s Reality-Virtuality Continuum
Mixed Reality
Reality - Virtuality (RV) Continuum
Real
Environment
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Environment

6. Metaverse

7. AR History Summary
 1960’s – 80’s: Early Experimentation
 1980’s – 90’s: Basic Research
 Tracking, displays
 1995 – 2005: Tools/Applications
 Interaction, usability, theory
 2005 - : Commercial Applications
 Games, Medical, Industry

8. Applications
 Medicine
 Manufacturing
 Information overlay
 Architecture
 Museum
 Marketing
 Gaming

9. AR Experience Design

11. “The product is no longer
the basis of value.The
experience is.”
Venkat Ramaswamy
The Future of Competition.

12. experiences
services
products
components
Value
Sony CSL © 2004
Gilmore + Pine: Experience Economy
Function
Emotion

13. The Value of Good User Experience
20c
50c
$3.50

14. Good Experience Design
 Reactrix
 Top down projection
 Camera based input
 Reactive Graphics
 No instructions
 No training

15. Apple: The Value of Good Design
 Good Experience Design Dominates Markets
iPod Sales 2002-2007

17. Using the N-gage

18. SideTalking
 http://www.sidetalkin.com

21. Interaction Design
“Designing interactive products to support people in their
everyday and working lives”
Preece, J., (2002). Interaction Design
 Design of User Experience with Technology
 Higher in the value chain than product design

23.  Interaction Design involves answering three questions:
 What do you do? - How do you affect the world?
 What do you feel? – What do you sense of the world?
 What do you know? – What do you learn?

24. Interaction Design is All About You
 Users should be
involved throughout
the Design Process
 Consider all the needs
of the user

27. Interaction Design Process

28. Gabbard Model for AR Design
1. user task analysis
2. expert guidelines-based evaluation
3. formative user-centered evaluation
4. summative comparative evaluations
Gabbard, J.L.; Swan, J.E.; , "Usability Engineering
for Augmented Reality: Employing User-Based
Studies to Inform Design,”
Visualization and Computer Graphics, IEEE
Transactions on, vol.14, no.3, pp.513-525, May-
June 2008

29. Gabbard Model in Context

30. experiences
applications
tools
components
Building Compelling AR Experiences
Tracking, Display
Authoring
Interaction
Usability

31. Summary
 In order to build AR applications you need to
focus on the user experience
 Great user experience is based on
 Low level AR component technology
 Authoring tools
 Application/Interaction design
 User experience texting

32. AR Technology

33. experiences
applications
tools
components
Sony CSL © 2004
Building Compelling AR Experiences
Display, Tracking

34. Core Technologies
 Combining Real and Virtual Images
• Display technologies
 Interactive in Real-Time
• Input and interactive technologies
 Registered in 3D
• Viewpoint tracking technologies
Display
Processing
Input Tracking

35. AR Displays

36. AR Displays
e.g. window
reflections
Virtual Images
seen off windows
e.g. Reach-In
Projection CRT Display
using beamsplitter
Not Head-Mounted
e.g. Shared Space
Magic Book
Liquid Crystal
Displays LCDs
Head-Mounted
Display (HMD)
Primarily Indoor
Environments
e.g. WLVA
and IVRD
Cathode Ray Tube (CRT)
or Virtual Retinal Display (VRD)
Many Military Applications
& Assistive Technologies
Head-Mounted
Display (HMD)
e.g. Head-Up
Display (HUD)
Projection Display
Navigational Aids in Cars
Military Airborne Applications
Not Head Mounted
(e.g. vehicle mounted)
Primarily Outdoor
(Daylight) Environments
AR
Visual Displays

37. Display Technologies
 Types (Bimber/Raskar 2003)
 Head attached
• Head mounted display/projector
 Body attached
• Handheld display/projector
 Spatial
• Spatially aligned projector/monitor

38. Display Taxonomy

39. Head Mounted Displays

40. Head Mounted Displays (HMD)
- Display and Optics mounted on Head
- May or may not fully occlude real world
- Provide full-color images
- Considerations
• Cumbersome to wear
• Brightness
• Low power consumption
• Resolution limited
• Cost is high?

41. Key Properties of HMD
 Field of View
 Human eye 95 degrees horizontal, 60/70 degrees vertical
 Resolution
 > 320x240 pixel
 Refresh Rate
 Focus
 Fixed/manual
 Power
 Size

42. Types of Head Mounted Displays
The image cannot be
displayed. Your
computer may not have
enough memory to open
the image, or the image
may have been
corrupted. Restart your
computer, and then open
the file again. If the red x
still appears, you may
have to delete the image
and then insert it again.
The
ima
ge
can
not
be
dis
play
ed.
You
r
co
mp
uter
Occluded
See-thru
Multiplexed

43. Immersive VR Architecture
Head!
Tracker
Host!
Processor
Data Base!
Model
Rendering!
Engine
Frame!
Buffer
head position/orientation
to network
Display!
Driver
Non see-thru!
Image source
& optics
virtual
object
Virtual
World

44. See-thru AR Architecture
Head!
Tracker
Host!
Processor
Data Base!
Model
Rendering!
Engine
Frame!
Buffer
head position/orientation
to network
Display!
Driver
see-thru!
combiner
Virtual Image
superimposed!
over real world
object
real world
Image source

45. Optical see-through head-mounted display
Virtual images
from monitors
Real
World
Optical
Combiners

46. Optical See-Through HMD

47. Epson Moverio Display
 Stereo see-through display
 960 x 540 pixels, 23 degree Field of View, 60 Hz
 $700 USD
 Android Powered

48. View Through Optical See-Through HMD

49. DigiLens
 www.digilens.com
 Compact HOE
 Solid state optics
 Switchable Bragg Grating
 Stacked SBG
 Fast switching
 Ultra compact

50. The Virtual Retinal Display
 Image scanned onto retina
 Commercialized through Microvision
 Nomad System - www.mvis.com

51. Strengths of optical see-through AR
 Simpler (cheaper)
 Direct view of real world
 Full resolution, no time delay (for real world)
 Safety
 Lower distortion
 No eye displacement
 but COASTAR video see-through avoids this

52. Video AR Architecture
Head!
Tracker
Host!
Processor
Graphics!
renderer
Digital!
Mixer
Frame!
Buffer
head position/orientation
to network
Display!
Driver
Non see-thru!
Image source
& optics
Head-mounted
camera aligned to
display optics
Video!
Processor
Video image
of real world
Virtual image
inset into
video of real
world

53. Video see-through HMD
Video
cameras
Monitors
Graphics
Combiner
Video

54. Video See-Through HMD

55. Video see-through HMD
MR Laboratory’s COASTAR HMD
(Co-Optical Axis See-Through Augmented Reality)
Parallax-free video see-through HMD

56. TriVisio
 www.trivisio.com
 Stereo video input
 PAL resolution cameras
 2 x SVGA displays
 30 degree FOV
 User adjustable convergence
 $6,000 USD

57. View Through a Video See-Through HMD

58. Vuzix Display
 www.vuzix.com
 Wrap 920
 $350 USD
 Twin 640 x 480 LCD displays
 31 degree diagonal field of view
 Weighs less than three ounces

59. Strengths of Video See-Through AR
 True occlusion
 Kiyokawa optical display that supports occlusion
 Digitized image of real world
 Flexibility in composition
 Matchable time delays
 More registration, calibration strategies
 Wide FOV is easier to support

60. Optical vs. Video AR Summary
 Both have proponents
 Video is more popular today?
 Likely because lack of available optical products
 Depends on application?
 Manufacturing: optical is cheaper
 Medical: video for calibration strategies

61. Eye multiplexed AR Architecture
Head!
Tracker
Host!
Processor
Data Base!
Model
Rendering!
Engine
Frame!
Buffer
head position/orientation
to network
Display!
Driver
Virtual Image
inset into!
real world scene
real world
Opaque!
Image source

62. Virtual Image ‘inset’ into real

63. Virtual Vision Personal Eyewear

64. Virtual image inset into real world

65. Google Glass

66. View Through Google Glass

68. What's Inside Google Glass?

69. Spatial/Projected AR

70. Spatial Augmented Reality
 Project onto irregular surfaces
 Geometric Registration
 Projector blending, High dynamic range
 Book: Bimber, Rasker “Spatial Augmented Reality”

71. Projector-based AR
Examples:
Raskar, MIT Media Lab
Inami, Tachi Lab, U. Tokyo
Projector
Real objects
with retroreflective
covering
User (possibly
head-tracked)

72. Example of projector-based AR
Ramesh Raskar, UNC, MERL

73. Example of projector-based AR
Ramesh Raskar, UNC Chapel Hill

74. The I/O Bulb
 Projector + Camera
 John Underkoffler, Hiroshi Ishii
 MIT Media Lab

75. Head Mounted Projector
 Head Mounted Projector
 Jannick Rolland (UCF)
 Retro-reflective Material
 Potentially portable

76. Head Mounted Projector
 NVIS P-50 HMPD
 1280x1024/eye
 Stereoscopic
 50 degree FOV
 www.nvis.com

77. HMD vs. HMPD
Head Mounted Display Head Mounted Projected Display

78. Pico Projectors
 Microvision - www.mvis.com
 3M, Samsung, Philips, etc

79. MIT Sixth Sense
 Body worn camera and projector
 http://www.pranavmistry.com/projects/sixthsense/

80. Other AR Displays

81. Video Monitor AR
Video
cameras Monitor
Graphics Combiner
Video
Stereo
glasses

82. Examples

83. Virtual Showcase
 Mirrors on a projection table
 Head tracked stereo
 Up to 4 users
 Merges graphic and real objects
 Exhibit/museum applications
 Fraunhofer Institute (2001)
 Bimber, Frohlich

84. Augmented Paleontology
Bimber et. al. IEEE Computer Sept. 2002

85. Alternate Displays
LCD Panel Laptop PDA

86. Handheld Displays
 Mobile Phones
 Camera
 Display
 Input

87. Other Types of AR Display
 Audio
 spatial sound
 ambient audio
 Tactile
 physical sensation
 Haptic
 virtual touch

88. Haptic Input
 AR Haptic Workbench
 CSIRO 2003 – Adcock et. al.

89. Phantom
 Sensable Technologies (www.sensable.com)
 6 DOF Force Feedback Device

90. AR Haptic Interface
 Phantom, ARToolKit, Magellan