The first lecture from my 2012 graduate course on Augmented Reality. Taught at the

1. COSC 426: Augmented Reality
Mark Billinghurst
mark.billinghurst@hitlabnz.org
July 11th 2012
Lecture 1: Introduction

2. Mark Billinghurst
 PhD Electrical Engineering
 University of Washington
 Interaction Design
 Museum experiences
 Tools for designers
 Augmented Reality
 Mobile AR, Evaluation,
 Multimodal Interfaces, Collaborative
 Collaboration
 Enhanced FtF and remote collaboration
 Social networking

3. Overview
 One two hour lecture a week
 Wednesday 1pm – 3pm
 You will learn
 Introduction to Augmented Reality
 Augmented Reality technology
 AR Interaction techniques
 Interaction Design
 AR authoring tools
 Research directions in AR
 Complete a simple project

4. Course Outline
 Wk 1 (July 11th): Introduction to Augmented Reality (AR)
 Wk 2 (July 18th): AR Technology
 Wk 3 (July 25th): AR Developer Tools
 Wk 4 (Aug 1st): AR Interaction Techniques
 Wk 5 (Aug 8th): AR Applications
 Wk 6 (Aug 15th): Outdoor and Mobile AR
 Wk 7,8 (Aug 22nd, Aug 29th): Holidays
 Wk 9 (Sept 5th): Collaborative AR
 Wk 10 (Sept 12th): Usability Testing
 Wk 11 (Sept 19th): AR research Directions
 Wk 12 (Sept 26th): Final Project Presentations

5. Assessment - Update
 Research project – 40%
 Group work (2-4 people)
 Due Sept 28th
 Two Class Assignments – 20 %
 Programming assignments, individual work
 Final Exam – 40%
 Exam week Oct 3rd – 14th

6. Introduction

7. A Brief History of Time
 Trend
 smaller, cheaper, more functions, more intimate
 Technology becomes invisible
 Intuitive to use
 Interface over internals
 Form more important than function
 Human centered design

8. A Brief History of Computing
 Trend
 smaller, cheaper, faster, more intimate, intelligent objects
 Computers need to become invisible
 hide the computer in the real world
- Ubiquitous / Tangible Computing
 put the user inside the computer
- Virtual Reality

9. Invisible Interfaces
Jun Rekimoto, Sony CSL

10. Graphical User Interfaces
 Separation between real and digital worlds
 WIMP (Windows, Icons, Menus, Pointer) metaphor

11. Ubiquitous Computing
 Computing and sensing embedded in real world
 Particle devices, RFID, motes, arduino, etc

12. Virtual Reality
 1985…

13. Virtual Reality
 Immersive VR
 Head mounted display, gloves
 Separation from the real world

14. 1977 – Star Wars

15. Augmented Reality Definition
 Defining Characteristics [Azuma 97]
 Combines Real and Virtual Images
- Both can be seen at the same time
 Interactive in real-time
- The virtual content can be interacted with
 Registered in 3D
- Virtual objects appear fixed in space

16. 2008 - CNN

17. Augmented Reality Examples
 Put AR pictures here

18. AR vs VR
 Virtual Reality: Replaces Reality
 Scene Generation: requires realistic images
 Display Device: fully immersive, wide FOV
 Tracking and Sensing: low accuracy is okay
 Augmented Reality: Enhances Reality
 Scene Generation: minimal rendering okay
 Display Device: non-immersive, small FOV
 Tracking and Sensing: high accuracy needed

19. Milgram’s Reality-Virtuality continuum
"...anywhere between the extrema of the virtuality continuum."
Mixed Reality
Real Augmented Augmented Virtual
Environment Reality (AR) Virtuality (AV) Environment
Reality - Virtuality (RV) Continuum
P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays
IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

20. Augmented Virtuality
 VR with windows into the real world

21. Metaverse
 Neal Stephenson’s “SnowCrash”
 The Metaverse is the convergence of:
 1) virtually enhanced physical reality
 2) physically persistent virtual space
 Metaverse Roadmap
 http://metaverseroadmap.org/

22. Metaverse Dimensions
• Augmentation technologies that layer information onto
our perception of the physical environment.
• Simulation refers to technologies that model reality
• Intimate technologies are focused inwardly, on the
identity and actions of the individual or object;
• External technologies are focused outwardly, towards
the world at large;

23. Metaverse Components
 Four Key Components
 Virtual Worlds
 Augmented Reality
 Mirror Worlds
 Lifelogging

25. Mirror Worlds
 Mirror worlds are informationally-enhanced
virtual models of the physical world.
 Google Earth, MS Street View, Google Maps

27. LifeLogging
 Technologies record and report the intimate
states and life histories of objects and users
 Nokia LifeBlog, Nike+, FitBits

28.  Steve Mann - LifeLogging

29. Gordon Bell: LifeLogging
1 TB to store 65 years of data

30. Summary
 Augmented Reality has three key features
 Combines Real and Virtual Images
 Interactive in real-time
 Registered in 3D
 AR can be classified alongside other technologies
 Invisible Interfaces
 Milgram’s Mixed Reality continuum
 MetaVerse

31. AR History

32. A Brief History of AR (1)
 1960’s: Sutherland / Sproull’s
first HMD system was see-
through

33. A Brief History of AR (2)
F16 – Head Up Display

34. A Brief History of AR (3)
1960 - 70’s: US Air Force helmet mounted
displays (T. Furness)

35. A Brief History of AR (4)
1970 - 80’s: US Air Force Super Cockpit (T. Furness)

36. A Brief History of AR (5)
 Early 1990’s: Boeing coined the term “AR.” Wire harness
assembly application begun (T. Caudell, D. Mizell).

37. A Brief History of AR (6)
 1994: Motion stabilized display [Azuma]
 1995: Fiducial tracking in video see-through [Bajura / Neumann]
 1996: UNC hybrid magnetic-vision tracker

38. A Brief History of AR (7)
 1996: MIT Wearable Computing efforts
 1998: Dedicated conferences begin (ISMAR)
 Late 90’s: Collaboration, outdoor, interaction
 Late 90’s: Augmented sports broadcasts

39. History Summary
 1960’s – 80’s: Early Experimentation
 1980’s – 90’s: Basic Research
 Tracking, displays
 1995 – 2005: Tools/Applications
 Interaction, usability, theory
 2005 - : Commercial Applications
 Games, Medical, Industry

40. 2007 - AR Reaches Mainstream
 MIT Technology Review
 March 2007
 list of the 10 most
exciting technologies
 Economist
 Dec 6th 2007
 Reality, only better

41. Gartner Hype Cycle

42. 2009 - AR in Magazines
 Esquire Magazine
 Dec 2009 issue
 12 pages AR content
 Many Others
 Wired
 Colors
 Red Bull
 Etc

43. Google Searches for AR

44. 2008 - Browser Based AR
 Flash + camera + 3D graphics
 High impact
 High marketing value
 Large potential install base
 1.6 Billion web users
 Ease of development
 Lots of developers, mature tools
 Low cost of entry
 Browser, web camera

47. Impact of Web-based AR
 Boffswana Living Sasquatch
 http://www.boffswana.com/news/?p=605
 In first month
 100K unique visits
 500K page views
 6 minutes on page

48. 2005 - Mobile Phone AR
 Mobile Phones
 camera
 processor
 display
 AR on Mobile Phones
 Simple graphics
 Optimized computer vision
 Collaborative Interaction

49. AR Advertising (HIT Lab NZ 2007)
 Txt message to download AR application (200K)
 See virtual content popping out of real paper advert
 Tested May 2007 by Saatchi and Saatchi

50. 2008: Location Aware Phones
Motorola Droid Nokia Navigator

51. 2009 - Outdoor Information Overlay
 Mobile phone based
 Tag real world locations
 GPS + Compass input
 Overlay graphics data on live video
 Applications
 Travel guide, Advertising, etc
 Wikitude, Layar, Junaio, etc..
 Android based, Public API released

52. Layar (www.layar.com)
 Location based data
 GPS + compass location
 Map + camera view
 AR Layers on real world
 Customized data
 Audio, 3D, 2D content
 Easy authoring
 Android, iPhone

53. $784 million USD in 2014

54. AR Today
 Key Technologies Available
- Robust tracking (Computer Vision, GPS/sensors)
- Display (Handheld HMDs)
- Input Devices (Kinect, etc)
- Developer tools (Qualcomm, Metaio, ARTW)
 Commercial Business Growing
- Gaming, GPS/Mobile, Online Advertisement
• >$5 Billion USD by 2016 (Markets andMarkets)
• >$1.5 Billion USD in Mobile AR by 2014 (Juniper Research)

55. AR Business Today
 Marketing
 Web-based, mobile
 Mobile AR
 Geo-located information and service
 Driving demand for high end phones
 Gaming
 Mobile, Physical input (Kinect, PS Move)
 Upcoming areas
 Manufacturing, Medical, Military

56. Some Commercial AR Companies
 ARToolworks (http://www.artoolworks.com/)
 ARToolKit, FLARToolKit, SDKs
 Metaio (http://www.metaio.com/)
 Marketing, Industry, SDKs
 Total Immersion (http://www.t-immersion.com/)
 Marketing, Theme Parks, AR Experiences
 Qualcomm (http://developer.qualcomm.com/dev/augmented-
reality)
 Mobile AR, Vuforia SDK
 Many small start-ups (String, Ogmento, etc)

57. Summary
 Augmented Reality has a long history going
back to the 1960’s
 Interest in AR has exploded over the last few
years and is being commercialized quickly
 AR is growing in a number of areas
 Mobile AR
 Web based AR
 Marketing experiences

58. Sample AR Applications

59. Applications
 Medicine
 Manufacturing
 Information overlay
 Architecture
 Museum
 Marketing
 Gaming

60. Applications: medical
 “X-ray vision” for surgeons
 Aid visualization, minimally-invasive operations.
Training. MRI, CT data.
 Ultrasound project, UNC Chapel Hill.
Courtesy
UNC
Chapel
Hill

61. Medical AR Trials
 Sauer et al. 2000 at Siemens
Corporate Research, NJ
 Stereo video see through
F. Sauer, Ali Khamene, S. Vogt: An Augmented Reality Navigation System with a
Single-Camera Tracker: System Design and Needle Biopsy Phantom Trial,
MICCAI 2002

62. Assembly and maintenance
© 1996 S. Feiner, B. MacIntyre, &
A. Webster, Columbia University
© 1993 S. Feiner, B. MacIntyre, &
D. Seligmann, Columbia University

63. PS3 - Eye of Judgment (2007)
 Computer Vision Tracking
 Card based battle game
 Collaborative AR
 October 24th 2007

65. AR Books – Markerless Tracking

66. AR Annotations
Columbia
University
© 1993 S. Feiner, B. MacIntyre, © 1997 S. Feiner, B. MacIntyre,
M. Haupt, & E. Solomon, T. Höllerer, & A. Webster,
Columbia University Columbia University
HRL

67. Broadcast TV

68. Interactive Museum Experiences
 BlackMagic
 Virtual America’s Cup
 410,000 people in six months
 MagicPlanet
 TeManawa science museum
 Virtual Astronomy
 Collaborative AR experience
 AR Volcano
 Interactive AR kiosk
 Scienceworks museum, Melbourne

69. Digital Binocular Station
http://www.DigitalBinocularStation.com/

70. Museum Archeology
 LifePlus (2002-2004)
 Natural feature tracking
 Virtual characters
 Mobile AR system
 Archeoguide (2000-2002)
 Cultural heritage on-site guide
 Hybrid tracking
 Virtual overlay

71. Sales and Marketing
 Connect with brands and branded objects
 Location Based Experiences
 Lynx Angels
 Web based
 Rayban glasses
 Mobile
 Ford Ka campaign
 Print based
 Red Bull Magazine

72. Summary
 AR technology can be used to develop a wide
range of applications
 Promising application areas include
 Games
 Education
 Engineering
 Medicine
 Museums
 Etc..

73. AR Experience Design

75. “The product is no longer
the basis of value. The
experience is.”
Venkat Ramaswamy
The Future of Competition.

76. Gilmore + Pine: Experience Economy
experiences Emotion
services
Value
products
Function
components
Sony CSL © 2004

77. The Value of Good User Experience
50c
$3.50
20c

78. Good Experience Design
 Reactrix
 Top down projection
 Camera based input
 Reactive Graphics
 No instructions
 No training

81. Using the N-gage

82. SideTalking
 http://www.sidetalkin.com

85. Interaction Design
“Designing interactive products to support people in their
everyday and working lives”
Preece, J., (2002). Interaction Design
 Design of User Experience with Technology
 Higher in the value chain than product design

87.  Interaction Design involves answering three questions:
 What do you do? - How do you affect the world?
 What do you feel? – What do you sense of the world?
 What do you know? – What do you learn?

88. Interaction Design is All About You
 Users should be
involved throughout
the Design Process
 Consider all the needs
of the user

91. Google Glasses

92. Interaction Design Process

93. Building Compelling AR Experiences
experiences
Usability
applications Interaction
tools Authoring
components Tracking, Display

94. Summary
 In order to build AR applications you need to
focus on the user experience
 Great user experience is based on
 Low level AR component technology
 Authoring tools
 Application/Interaction design
 User experience texting