1. COSC 426: Augmented Reality
426 A d R li
Mark Billinghurst
mark.billinghurst@hitlabnz.org
July 2011
Lecture 1

2. Mark Billinghurst
PhD Electrical Engineering
University of Washington
Interaction Design
Museum experiences
Tools for designers
Augmented Reality
Mobile AR, Evaluation
AR Evaluation,
Multimodal Interfaces, Collaborative
Collaboration
Enhanced FtF and remote collaboration
Social networking

3. Overview
One tow hour lectures week
Friday 9 – 11am
You will learn
Introduction to Augmented Reality
Augmented Reality technology
AR Interaction techniques
Interaction Design
AR authoring tools
Research directions in AR
Complete a simple project

4. Course Outline
C O l
Wk 1 (July 15th): Introduction to AR
(J y )
Wk 2 (July 22nd): AR Technology
Wk 3 (July 29th): AR Developer Tools
(J y ) p
Wk 4 (Aug 5th): AR Interaction Techniques
Wk 5 (Aug 12th): AR Applications
Wk 6 (Aug 19th): Outdoor and Mobile AR
Wk 7,8 (Aug 26th, Sept 2nd): Holidays
Wk 9 (Sept 9th): Collaborative AR
Wk 10 (Sept 16th): Usability Testing
Wk 11 (Sept 23rd): AR research Directions
Wk 12 (Sept 30th): Final Project Presentations

5. Assessment - Update
Research project – 40%
Group work (2-4 people)
Due Sept 30th
Two Class Assignments – 20 %
Programming assignments, individual work
g g g ,
Final Exam – 40%
Exam week Oct 3rd – 14th
E kO d

6. Introduction

7. A Brief History of Time
Trend
T d
smaller, cheaper, more functions, more intimate
Technology b
T h l becomes invisible
bl
Intuitive to use
Interface over internals
I f i l
Form more important than function
Human centered design
H t dd i

8. A Brief History of Computing
B fH fC
Trend
smaller, cheaper, faster, more intimate, intelligent objects
Computers need to become invisible
hide the computer in the real world
- Ubiquitous / Tangible Computing
put the user inside the computer
- Vi
Virtual R li
l Reality

9. Invisible Interfaces
Jun Rekimoto, Sony CSL

10. Virtual Reality
1989…
1989

12. Cheap HMDs

13. Virtual Reality
Immersive VR
Head mounted display, gloves
Separation from the real world

14. VR Today
$3-5 Billion VR business (+ > $150 Billion Graphics Industry)
Visualization, simulation, gaming, CAD/CAE, multimedia, graphics arts

15. 1977 – Star Wars

16. 2008 - CNN

17. Augmented Reality Definition
Defining Characteristics [Azuma 97]
D f Ch [A
Combines Real and Virtual Images
- Both can be seen at the same time
Interactive in real-time
- The virtual content can be interacted with
Registered in 3D
g
- Virtual objects appear fixed in space

18. Augmented Reality Examples
Put AR pictures here

19. AR vs VR
Virtual Reality: Replaces Reality
Scene Generation: requires realistic images
q g
Display Device: fully immersive, wide FOV
Tracking and Sensing: low accuracy is okay
Augmented Reality: Enhances Reality
Scene Generation: minimal rendering okay
Display Device: non-immersive small FOV
non immersive,
Tracking and Sensing: high accuracy needed

20. Milgram’s Reality-Virtuality continuum
Mixed Reality
Real Augmented Augmented Virtual
Environment Reality (AR) Virtuality (AV) Environment
Reality - Virtuality (RV) Continuum

21. Metaverse
Neal Stephenson’s “SnowCrash”
The Metaverse is the convergence of:
1) virtually enhanced physical reality
2) physically persistent virtual space
h ll l
Metaverse Roadmap
p
http://metaverseroadmap.org/

22. Metaverse Dimensions
• Augmentation technologies that layer information onto
our perception of the physical environment.
• Simulation refers to technologies that model reality
• Intimate technologies are focused inwardly, on the
identity and actions of the individual or object;
• External technologies are focused outwardly, towards
outwardly
the world at large;

23. Metaverse Components
Four Key Components
Virtual Worlds
Augmented Reality
Mirror Worlds
Lifelogging

25. Mirror Worlds
Mirror worlds are informationally-enhanced
virtual models of the physical world.
p y
Google Earth, MS Street View, Google Maps

27. LifeLogging
Technologies record and report the intimate
states and life histories of objects and users
j
Nokia LifeBlog, Nike+, FitBits

28. Steve Mann - LifeLogging
gg g

29. Gordon Bell: LifeLogging
1 TB to store 65 years of data

30. Summary
Augmented Reality has three key features
Combines Real and Virtual Images
g
Interactive in real-time
Registered in 3D
AR can be classified alongside other technologies
Milgrams Mixed Reality continuum
MetaVerse

31. AR History

32. A Brief History of AR (1)
1960’s: Sutherland Sproull’s
1960’ S h l d / S ll’
first HMD system was see-
through

33. A Brief History of AR (2)
1960 - 70’s: US Air Force helmet mounted
displays (T. Furness)
(T

34. A Brief History of AR (3)
1970 - 80’s: US Air Force Super Cockpit ( Furness)
p p (T. )

35. A Brief History of AR (4)
Early 1990’s: Boeing coined the term “AR.” Wire harness
990’ “A ”
assembly application begun (T. Caudell, D. Mizell).
Early t
E l to mid 1990’s: UNC ultrasound visualization project
id 1990’ lt d i li ti j t

36. A Brief History of AR (5)
1994: Motion stabilized display [Azuma]
1995: Fiducial tracking in video see-through [Bajura / Neumann]
1996: UNC hybrid magnetic-vision tracker
y g

37. A Brief History of AR (6)
1996: MIT Wearable Computing efforts
1998: Dedicated conferences begin
Late 90’s: Collaboration, outdoor, i
L 90’ C ll b i d interaction
i
Late 90’s: Augmented sports broadcasts
1998 - 2001: Mixed Reality Systems Lab

38. History Summary
1960’s – 80’s: Early Experimentation
1980 s 90 s:
1980’s – 90’s: Basic Research
Tracking, displays
1995 – 2005: Tools/Applications
Interaction, usability, theory
y y
2005 - : Commercial Applications
Games, M di l Industry
G Medical, I d

39. Gartner Hype Cycle

40. 2007 - AR Reaches Mainstream
MIT Technology Review
March 2007
list of the 10 most
exciting technologies
Economist
Dec 6th 2007
Reality, only better

41. Gartner’s top 10 disruptive technologies 2008-2012:
Multicore and hybrid processors
Virtualisation and fabric computing
Social
S i l networks and social software
k d i l f
Cloud computing and cloud/Web platforms
Web mash s
mashups
User Interface
Ubiquitous computing
Contextual computing
Augmented reality
Semantics

42. 2009 - AR in Magazines
Esquire Magazine
Dec 2009 issue
12 pages AR content
Many Others
M Oth
Wired
Colors
Red Bull
Etc

43. Google Searches for AR

44. 2008 - Browser Based AR
Flash + camera + 3D graphics
High impact
g p
High marketing value
Large potential install base
1.6 Billion web users
Ease of development
Lots of developers, mature tools
Low cost of entry
L f
Browser, web camera

47. Impact of Web-based AR
Boffswana Living Sasquatch
http://www.boffswana.com/news/?p=605
p p
In first month
100K unique visits
500K page views
6 minutes on page

48. Mobile Phone AR
Mobile Phones
camera
processor
display
d l
AR on Mobile Phones
Simple graphics
Optimized computer vision
Collaborative Interaction

49. AR Advertising (HIT Lab NZ 2007)
Txt
T message to d download AR application (200K)
l d li i
See virtual content popping out of real paper advert
Tested May 2007 by Saatchi and Saatchi

50. 2008: Location Aware Phones
Motorola Droid Nokia Navigator

51. 2009 - Outdoor Information Overlay
Mobile h
M bil phone based
b d
Tag real world locations
g
GPS + Compass input
Overlay graphics data on live video
Applications
Travel guide, Advertising, etc
Wikitude, Layar, Junaio, etc..
, y ,J ,
Android based, Public API released

52. Layar (www.layar.com)
Location based data
GPS + compass location
p
Map + camera view
AR Layers on real world
L l ld
Customized data
Audio, 3D, 2D content
Easy authoring
Android, iPhone

53. $784 million USD in 2014

54. Some Commercial AR Companies
ARToolworks (http://www artoolworks com/)
(http://www.artoolworks.com/)
ARToolKit, FLARToolKit, SDKs
Metaio (http://www.metaio.com/)
Marketing, Industry, SDKs
Total Immersion (http://www.t-immersion.com/)
Marketing, Theme Parks, AR Experiences
Qua co
Qualcomm
(http://developer.qualcomm.com/dev/augmented-reality)
Mobile AR QCAR SDK
AR,

55. FLARToolKit (ARToolworks)
Flash version of ARToolKit
FLARToolKit + Papervison3D = AR on the web
Example - GE Ecomagination Website
1.2 million views on YouTube
http://ge.ecomagination.com/smartgrid/#/augmented_reality
p g g g g y

56. Software Platform (Metaio)
Unifeye software platform
Modular platform for building AR applications
p g pp

57. Markerless Tracking (Total Immersion)
D’Fusion markerless tracking platform
Turn key solution
y
Example - Topps baseball cards
AR characters on top of real baseball cards
h f l b b ll d

58. Summary
Augmented Reality has a long history going
back to the 1960’s
Interest in AR has exploded over the last two
years and is being commercialized quickly
AR is growing in a number of areas
Mobile AR
Web based AR
Advertising experiences

59. Sample AR Applications

60. Applications
Medicine
Manufacturing
Information overlay
Architecture
Museum
Marketing
Gaming

61. Applications: medical
“X-ray vision” for surgeons
Aid visualization, minimally-invasive operations.
minimally invasive
Training. MRI, CT data.
Ultrasound project, UNC Ch l Hill
Ul d j Chapel Hill.
Courtesy
UNC
Chapel
Hill

62. Medical AR Trials
Sauer et al. 2000 at Siemens
Corporate Research, NJ
Stereo video see through
F. Sauer, Ali Khamene, S. Vogt: An Augmented Reality Navigation System with a
Single Camera
Single-Camera Tracker: System Design and Needle Biopsy Phantom Trial,
MICCAI 2002

64. Assembly and maintenance
© 1996 S. Feiner, B. MacIntyre, &
A. Webster, Columbia University
© 1993 S F i
S. Feiner, B M I t
B. MacIntyre, &
D. Seligmann, Columbia University

65. PS3 - Eye of Judgment (2007)
Computer Vi i T ki
C t Vision Tracking
Card based battle game
Collaborative
C ll b ti AR
October 24th 2007

67. AR Books – Markerless Tracking

68. AR Jam story book

69. AR A
Annotations
t ti
Columbia
University
© 1993 S. Feiner, B. MacIntyre, © 1997 S. Feiner, B. MacIntyre,
M. Haupt, & E. Solomon, T. Höllerer, & A. Webster,
, ,
Columbia U i
C l bi University i Columbia University
HRL

70. Applications: Annotating Environment
Public d i
P bli and private annotations
i
Aid recognition, “extended memory”
g , y

71. The World = Your User Interface (Feiner)
Everyone, Everywhere, Always
E E h Al
© 2002, Scientific American

72. Application: Broadcast Augmentation
Adding virtual content to live sports broadcasts
“First down” line in American football
First down
Hockey puck trails, virtual advertisements
National flags in swimming lanes in 2000 Olympics
Commercial application
Princeton Video Image is one company
- http://www.pvi.tv/

73. Broadcast Examples

74. Interactive Museum Experiences
BlackMagic
Virtual America’s Cup
410,000 people in six months
MagicPlanet
TeManawa science museum
Virtual Astronomy
Collaborative AR experience
AR Volcano
Interactive AR kiosk
Scienceworks museum, Melbourne

75. Museum Archeology
LifePlus (2002-2004)
Natural feature tracking
g
Virtual characters
Mobile AR system
Archeoguide (2000-2002)
Cultural heritage on-site guide
Hybrid tracking
Virtual overlay

76. Summary
AR technology can b used to develop a wide
h l be d d l d
range of applications
Promising application areas include
Games
Education
Engineering
Medicine
Museums
M
Etc..

77. AR Experience Design

78. “The product is no longer
The
the basis of value The
value.
experience is ”
is.
Venkat Ramaswamy
The Future of Competition.

79. Gilmore + Pine: Experience Economy
experiences Emotion
E ti
services
products
components Function
Sony CSL © 2004

80. The Value of Good User Experience
50c
50
$3.50
20c

81. Good Experience Design
Reactrix
Top down projection
Camera based input
Reactive Graphics
No instructions
No training

84. Using the N-gage

85. SideTalking
http://www.sidetalkin.com

88. Interaction Design
“Designing interactive products to support people in their
everyday and working lives”
lives
Preece, J., (2002). Interaction Design
Design of User Experience with Technology
Higher in the value chain than product design

91. Interaction Design involves answering three questions:
What do you do? - How do you affect the world?
What do you feel? – What do you sense of the world?
What do you know? – What do you learn?

92. Interaction Design Process

93. Who are your Users?
Everyone!
y

94. Designing for Everyone
Designing for Everyone pleases No one

95. Understanding Specific Needs

97. Interaction Design is All About You
Users should be
involved throughout
the Design Process
Co s de all the eeds
Consider a t e needs
of the user

100. Building Compelling AR Experiences
B ildi C lli E i
experiences
Usability
applications Interaction
tools Authoring
components Tracking, Display

101. Summary
In order to build AR applications you need to
p
focus on the user experience
Great user experience is based on
Low l l AR component technology
L level h l
Authoring tools
Application/Interaction design
User experience texting