This document provides an overview of a presentation on designing compelling augmented reality (AR) and virtual reality (VR) experiences. The presentation will cover definitions of AR and VR, example applications, hands-on experience with authoring tools ENTiTi Creator and Wikitude World, and research directions. It will also discuss challenges in designing experiences for AR and VR head-mounted displays using mobile devices as computing modules.
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
Â
AR-VR Workshop
1. DESIGNING COMPELLING
AR AND VR EXPERIENCES
Mark Billinghurst
mark.billinghurst@unisa.edu.au
Zi Siang See
zisiangsee@sunway.edu.my
vsmm2016.org
October 17th 2016
2. About Us
• Mark
• PhD University of Washington
• Founder, HIT Lab NZ
• Professor, University South Australia
• Zi Siang
• Faculty member, Sunway
• Creative Director, Reina Imaging
• Academic, University Tunku Abdul Rahman
3. Overview
• 9:30 Introduction (Mark + Zi Siang)
• 9:35 Introduction to Virtual Reality (Mark)
• 10:00 Developing VR with ENTiTi (Zi Siang)
• 10:30 Introduction to Augmented Reality (Mark)
• 11:00 Developing AR with ENTiTi (Zi Siang)
• 11:30 Building Outdoor AR with Wikitude (Zi Siang)
• 12:00 Research Directions/Questions (Mark)
• 12:30 Finish
4. What You Will Learn
• Definitions of AR/VR
• History of AR/VR
• Example applications
• How to make AR/VR experiences
• Hands-on with authoring tools
• Best interaction methods
• Research directions in AR/VR
5. Authoring Tools Used
1)  Augmented Reality (AR)
2)  Virtual Reality (VR)
3)  Outdoor AR experience
- ENTiTi Creator
- ENTiTi Creator
- Wikitude World
6. ENTiTi Creator (Desktop)
• AR/VR application building for non-programmers
• Available from http://www.wakingapp.com
Install for PC or Mac
12. The Ultimate Display
“The ultimate display would, of course, be a room
within which the computer can control the
existence of matter. A chair displayed in such a
room would be good enough to sit in. Handcuffs
displayed in such a room would be confining, and
a bullet displayed in such a room would be fatal”.
Ivan Sutherland, 1965
16. What is Virtual Reality?
Virtual reality is..
a computer technology that replicates an
environment, real or imagined, and simulates a
user's physical presence and environment to
allow for user interaction. (Wikipedia)
• Defining Characteristics
• Environment simulation
• Presence
• Interaction
36. •  In 2016 – 46m possible desktop VR users vs. 400 m mobile VR users
•  https://thoughts.ishuman.co/vr-will-be-mobile-11529fabf87c#.vfcjzy1vf
44. Google Cardboard App
• 7 default experiences
•  Earth: Fly on Google Earth
•  Tour Guide: Visit sites with guides
•  YouTube: Watch popular videos
•  Exhibit: Examine cultural artifacts
•  Photo Sphere: Immersive photos
•  Street View: Drive along a street
•  Windy Day: Interactive short story
49. What You Need
• Cardboard Viewer/VR Viewer
•  https://www.google.com/get/cardboard/
• Smart phone
•  Android/iOS
• Authoring Tools/SDK
•  Google VR SDK
•  Unity/Unreal game engine
•  Non programming tools
• Content
•  3D models, video, images, sounds
50. Software Tools
• Low level SDKs
•  Need programming ability
•  Java, C#, C++, etc
•  Example: Google VR SDK (iOS, Android)
•  https://developers.google.com/vr/
• Game Engines
•  Powerful, need scripting ability
• Unity - https://unity3d.com/
• Unreal - https://www.unrealengine.com/vr
•  Combine with VR plugins (HMDs, input devices)
•  Google VR Unity plugin
52. Tools for Non-Programmers
•  Focus on Design, ease of use
•  Visual Programming, content arrangement
•  Examples
•  Insta-VR – 360 panoramas
•  http://www.instavr.co/
•  Vizor – VR on the Web
•  http://vizor.io/
•  A-frame – HTML based
•  https://aframe.io/
•  ENTiTi – Both AR and VR authoring
•  http://www.wakingapp.com/
•  Eon Creator – Drag and drop tool for AR/VR
•  http://www.eonreality.com/eon-creator/
53. Designing Mobile VR Applications
• Things to consider
•  Ease of use
•  Type of experience
•  Immersive images vs. 3d interaction
•  Length of experience
•  2D versus 3D information presentation
• Constraints
•  Limited graphics power
•  Limited user input/interaction
•  Head pointing, button
•  Limited feedback (audio, video, no haptic)
54. Physiological Considerations
• Factors to Consider
•  Head tracking
•  User control of movement
•  Use constant velocity
•  Grounding with fixed objects
•  Brightness changes
55. Universal VR Interaction Tasks
• Object Interaction
•  Selection: Picking object(s) from a set
•  Manipulation: Modifying object properties
• Navigation
•  Travel: motor component of viewpoint motion
•  Wayfinding: cognitive component; decision-making
• System control
•  Issuing a command to change system state or mode
59. Interactive Patterns - Display Reticle
•  Easier for users to target objects with a display reticle
•  Can display reticle only when near target object
•  Highlight objects (e.g. with light source) that user can target
61. Example: Gaze Menu Selection
https://www.youtube.com/watch?v=T0PNfc_Yibk
62. Interactive Patterns - Controls
• Fuze buttons
•  Time based head pointing with no click input
•  Visual countdown, button placement
• Gaze and click
•  Target size and selection
67. Navigation: Gaze Directed Walking
•  Move in direction that you are looking
•  Very intuitive, natural navigation
•  Can be used on simple HMDs (Google Cardboard
•  But: Can’t look in different direction while moving
82. AR & VR on Head Mount Devices
Mobile device
(as computing module)
83. •  Carl Zeiss just announced VR One, a virtual reality headset for use with a
smartphone. It is a viewer designed to work with phones between 4.7 and 5.2
inches (Zeiss, 2014).
Google Cardboard
84. •  Carl Zeiss just announced VR One, a virtual reality headset for use with a
smartphone. It is a viewer designed to work with phones between 4.7 and 5.2
inches (Zeiss, 2014).
Programmable
NFC Tag
Low cost resin /
plastic lens
Mobile device as
computing module
Endless configurable size
and shapes (and materials)
Magnetometer
Google Cardboard (v1)
85. more reading at http://www.gizmag.com/google-cardboard-2-review-initial/37777/
Presses the
screen
Figure: google cardboard v2 supports larger phone.
Google Cardboard (v2)
89. Challenges
•  Consideration
•  Mostly everyone has a mobile device
•  Only few individuals have expensive HMD
•  Everyone can access to low-cost HMD
•  Design and Development
•  Content creation, authoring
•  Usable user interface (minimal?)
•  Display platforms or systems
•  Ease-of-use
•  Useful Experience
90. Local or Cloud-based App?
Cloud-based
•  Unlimited numbers of recognition
•  Unlimited content from server
•  Requires network
•  Stability relies on network speed
•  OS update safe for content
•  Shows new content automatically
•  Users does not own content
Local-based
•  Limited numbers of recognition
•  Limited content in-App
•  Works offline
•  Stable
•  OS update affects App & content
•  Requires users to update App
•  Users can own content
91. Authoring
•  Virtual Reality
•  Virtual Reality (VR) which can be referred to as
immersive multimedia or computer-simulated life,
replicates an environment that simulates physical
presence in places in the real world or imagined worlds
and lets the user interact in that world.
•  Augmented Reality
•  Augmented Reality (AR) mixes a live real-world view
with virtual interactive content on a mobile or wearable
device. One of the key enablers for this is tracking
technology, such as computer vision techniques for
tracking off pre-defined markers or markerless images.
128. Visual Programming
• Benefits
• Minimum coding required
• Prototyping basic VR scenario
• Recommendation
• Storyline of scenario is essential
• Content quality requires lengthy development
• Choose suitable platforms for specific need
131. Augmented Reality Definition
• Defining Characteristics [Azuma 97]
• Combines Real andVirtual Images
• Both can be seen at the same time
• Interactive in real-time
• The virtual content can be interacted with
• Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
135. Milgram’s Reality-Virtuality continuum
Mixed Reality
Reality - Virtuality (RV) Continuum
Real
Environment
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Environment
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays
IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.
140. Industrial andAcademic Research (1990’s- )
• Early 1990’s: Boeing coined the term “AR.”
• Mid 1990’s AR research in tracking and display (UNC)
141. Early Commercialization (2000 – 2010)
• 2000:Augmented sports broadcasts
• 2007: PlayStation Eye of Judgement
142. Consumer Adoption (2009 - )
•  Web pages with AR experiences integrated into them
•  Smart phones with built-in sensors suitable for mobile AR
144. 2005 - Mobile PhoneAR
• Mobile Phones
• camera
• processor
• display
• AR on Mobile Phones
• Simple graphics
• Optimized computer vision
• Collaborative Interaction
145. ARAdvertising (HIT Lab NZ 2007)
• Txt message to download AR application (200K)
• See virtual content popping out of real paper advert
• Tested May 2007 by Saatchi and Saatchi
147. 2009 - Outdoor Information Overlay
• Mobile phone based
• Tag real world locations
• GPS + Compass input
• Overlay graphics on live video
• Applications
• Travel guide,Advertising, etc
• Wikitude, Layar, etc..
• iOS/Android, Public API released
149. Augmented Reality BusinessToday
• Rapidly Growing
• > $80 Billion USD by 2020
• Wide range of HW/SW available
• HMD, mobile phones, PCs
• Many easy to use developer tools
• Many application areas
• Marketing, gaming, education
• Mobile AR
151. Pokemon GO Effect
•  Fastest App to reach $500 million in Revenue
•  Only 63 days after launch, > $1 Billion in 6 months
•  Over 500 million downloads, > 25 million DAU
•  Nintendo stock price up by 50% (gain of $9 Billion USD)
161. •  Weak AR
•  Imprecise tracking
•  No knowledge of environment
•  Limited interactivity
•  Handheld AR
•  Strong AR
•  Very accurate tracking
•  Seamless integration into real world
•  Natural interaction
•  Head mounted AR
Strong vs. Weak AR
166. •  Web based AR
•  Flash, HTML 5 based AR
•  Marketing, education
•  Outdoor Mobile AR
•  GPS, compass tracking
•  Viewing Points of Interest in real world
•  Handheld AR
•  Vision based tracking
•  Marketing, gaming
•  Location Based Experiences
•  HMD, fixed screens
•  Museums, point of sale, advertising
Typical AR Experiences
175. Education:Quiver (iOS/Android)
•  Interactive Colouring Books
•  Children colour their own AR scenes
•  Wide range of educational pages available
•  Animals, cells, volcanos, etc
•  http://www.quivervision.com/
182. TangibleAR Interaction Metaphor
• AR overcomes limitation of TUIs
• enhance display possibilities
• merge task/display space
• provide public and private views
• TUI + AR = Tangible AR
• Apply TUI methods to AR interface design
183. TangibleAR Design Principles
• Tangible AR Interfaces use TUI principles
• Physical controllers for moving virtual content
• Support for spatial 3D interaction techniques
• Time and space multiplexed interaction
• Support for multi-handed interaction
• Match object affordances to task requirements
• Support parallel activity with multiple objects
• Allow collaboration between multiple users
185. Handheld HCI
• Consider your user
• Follow good HCI principles
• Adapt HCI guidelines for handhelds
• Design to device constraints
• Rapid prototyping
• User evaluation
186. ConsiderYour User
• Consider context of user
•  Physical, social, emotional, cognitive, etc
• Mobile Phone AR User
•  Probably Mobile
•  One hand interaction
•  Short application use
•  Need to be able to multitask
•  Use in outdoor or indoor environment
•  Want to enhance interaction with real world
187. Applying Principles to MobileAR
•  Clean
•  LargeVideoView
•  Large Icons
•  Text Overlay
•  Feedback
188. Design to Device Constraints
• Understand the platform and design for limitations
• Hardware, software platforms
• Eg Handheld AR game with visual tracking
• Use large screen icons
• Consider screen reflectivity
• Support one-hand interaction
• Consider the natural viewing angle
• Do not tire users out physically
• Do not encourage fast actions
• Keep at least one tracking surface in view
Art of Defense Game
190. Handheld Interface Metaphors
•  Tangible AR LensViewing
•  Look through screen into AR scene
•  Interact with screen to interact with AR
content
•  Eg Invisible Train
•  Tangible AR Lens Manipulation
•  Select AR object and attach to device
•  Use the motion of the device as input
•  Eg AR Lego
201. ENTiTi Creator
1)  Assign your image
2)  Click “Preview”
“OK” for including new video
Sample is provided “entiti_ar_showreel_small.mp4”
214. Wikitude World: location-based AR
•  KML
•  Keyhole Markup Language is a standardized format
used in Google Earth. It can provide basic information
for POIs and easily uploaded (or using hyperlink) into
Wikitude.
•  KML files can be created with the Google Earth user
interface (in Google Map)
•  Alternatively, it can be created using XML or simple text
editor to work on raw KML scripts from scratch.
https://developers.google.com/kml/documentation/?hl=en
215. Wikitude World: location-based AR
•  ARML
•  Augmented Reality Markup Language is an open
exchange format based on KML but extends the format
for useful data around AR data sets.
•  ARML 2.0 is used in the live versions of the 3 leading
Augmented Reality Browsers (Junaio, Layar and
Wikitude), where it is used to make the AR Browsers
interoperable.
•  ARML 1.0 file format is currently used in the Wikitude
World Browser.
http://openarml.org/wikitude4.html
216. Steps for Using Wikitude
1.  Register for Wikitude account
•  http://www.wikitude.com
2.  Register for web hosting account
•  http://www.000webhost.com
•  Own domain and server hosting
3.  Content authoring
•  https://www.google.com/mymaps/
•  http://studio.wikitude.com
•  Scripting and customization
227. Steps for Setting up Wikitude
1.  Register for Wikitude account
•  http://www.wikitude.com
2.  Register for web hosting account
•  http://www.000webhost.com
•  Or use own domain and server hosting
229. KML
•  Benefits
•  Generated from Google Earth
•  Allows basic editing
•  Current limitation
•  Limited scripting option for Wikitude World
•  Lack of options for POI details in Wikitude World
•  Range is confined to ~20km radius
247. Publish in Wikitude: ARML
• KML
• [put stuff here]
• ARML
• [put stuff here]
provide URL, host the *.xml file
from your own server
249. ARML
•  Benefits
•  Extended scripting option for Wikitude World
•  Additional POI details in Wikitude World
•  Current limitation
•  Propagation: Wikitude World may take time to be
visible.
•  Range is confined to ~20km radius
•  Future possibilities
•  AR link may be interoperated with other AR browser.
273. Evolution of Tracking
• Past
•  Location based, marker based,
•  magnetic/mechanical
• Present
•  Image based, hybrid tracking
• Future
•  Ubiquitous
•  Model based
•  Environmental
274. Model Based Tracking (1-3 yrs)
• Track from known 3D model
•  Use depth + colour information
•  Match input to model template
•  Use CAD model of targets
• Recent innovations
•  Learn models online
•  Tracking from cluttered scene
•  Track from deformable objects
Hinterstoisser, S., Lepetit, V., Ilic, S., Holzer, S., Bradski, G., Konolige, K., & Navab, N. (2013).
Model based training, detection and pose estimation of texture-less 3D objects in heavily
cluttered scenes. In Computer Vision–ACCV 2012 (pp. 548-562). Springer Berlin Heidelberg.
276. Environmental Tracking (3+ yrs)
• Environment capture
•  Use depth sensors to capture scene & track from model
• InifinitAM (www.robots.ox.ac.uk/~victor/infinitam/)
•  Real time scene capture on mobiles, dense or sparse capture
•  Dynamic memory swapping allows large environment capture
•  Cross platform, open source library available
280. Wide Area Outdoor Tracking (5+ yrs)
• Process
•  Combine panorama’s into point cloud model (offline)
•  Initialize camera tracking from point cloud
•  Update pose by aligning camera image to point cloud
•  Accurate to 25 cm, 0.5 degree over very wide area
Ventura, J., & Hollerer, T. (2012). Wide-area scene mapping for mobile visual tracking. In Mixed
and Augmented Reality (ISMAR), 2012 IEEE International Symposium on (pp. 3-12). IEEE.
281. Wide Area Outdoor Tracking
https://www.youtube.com/watch?v=8ZNN0NeXV6s
282. Outdoor Localization using Maps
•  Use 2D building footprints and approximate height
•  Process
•  Sensor input for initial position orientation
•  Estimate camera orientation from straight line segments
•  Estimate camera translation from façade segmentation
•  Use pose estimate to initialise SLAM tracking
•  Results – 90% < 4m position error, < 3
o
angular error
Arth, C., Pirchheim, C., Ventura, J., Schmalstieg, D., & Lepetit, V. (2015). Instant outdoor
localization and SLAM initialization from 2.5 D maps. IEEE transactions on visualization and
computer graphics, 21(11), 1309-1318.
286. Natural Gesture (2-5 years)
• Freehand gesture input
•  Depth sensors for gesture capture
•  Move beyond simple pointing
•  Rich two handed gestures
• Eg Microsoft Research Hand Tracker
•  3D hand tracking, 30 fps, single sensor
• Commercial Systems
•  Meta, MS Hololens, Occulus, Intel, etc
Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Leichter, D. K. C. R. I., ... & Izadi, S.
(2015, April). Accurate, Robust, and Flexible Real-time Hand Tracking. In Proc. CHI (Vol. 8).
288. Multimodal Input (5+ years)
• Combine gesture and speech input
•  Gesture good for qualitative input
•  Speech good for quantitative input
•  Support combined commands
•  “Put that there” + pointing
• Eg HIT Lab NZ multimodal input
•  3D hand tracking, speech
•  Multimodal fusion module
•  Complete tasks faster with MMI, less errors
Billinghurst, M., Piumsomboon, T., & Bai, H. (2014). Hands in Space: Gesture Interaction with
Augmented-Reality Interfaces. IEEE computer graphics and applications, (1), 77-80.
289. HIT Lab NZ Multimodal Input
https://www.youtube.com/watch?v=DSsrzMxGwcA
290. Intelligent Interfaces (10+ years)
• Move to Implicit Input vs. Explicit
•  Recognize user behaviour
•  Provide adaptive feedback
•  Support scaffolded learning
•  Move beyond check-lists of actions
• Eg AR + Intelligent Tutoring
•  Constraint based ITS + AR
•  PC Assembly (Westerfield (2015)
•  30% faster, 25% better retention
Westerfield, G., Mitrovic, A., & Billinghurst, M. (2015). Intelligent Augmented Reality Training for
Motherboard Assembly. International Journal of Artificial Intelligence in Education, 25(1), 157-172.
298. Example:Visualizing Sensor Networks
•  Rauhala et. al. 2007 (Linkoping)
•  Network of Humidity Sensors
•  ZigBee wireless communication
•  Use Mobile AR toVisualize Humidity
304. Example: Social Panoramas
• Google Glass
• Capture live image panorama (compass + camera)
• Remote device (tablet)
• Immersive viewing, live annotation
Reichherzer, C., Nassani, A., & Billinghurst, M. (2014). Social panoramas using wearable
computers. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on
(pp. 303-304). IEEE.
306. Empathy Glasses (CHI 2016)
•  Combine together eye-tracking, display, face expression
•  Impicit cues – eye gaze, face expression
++
Pupil Labs Epson BT-200 AffectiveWear
Masai, K., Sugimoto, M., Kunze, K., & Billinghurst, M. (2016, May). Empathy Glasses. In Proceedings of
the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM.
307. AffectiveWear – Emotion Glasses
•  Photo sensors to recognize expression
•  User calibration
•  Machine learning
•  Recognizing 8 face expressions
308. Remote Collboration
• Eye gaze pointer and remote pointing
• Face expression display
• Implicit cues for remote collaboration
311. Example:SocialAcceptance
• People don’t want to look silly
•  Only 12% of 4,600 adults would be willing to wear AR glasses
•  20% of mobile AR browser users experience social issues
• Acceptance more due to Social than Technical issues
•  Needs further study (ethnographic, field tests, longitudinal)
317. AR + Smart Sensors + Social Networks
• Track population at city scale (mobile networks)
• Match population data to external sensor data
• Mine data for applications
318. Example: MIT SENSEable City Lab
https://www.youtube.com/watch?v=eXOCbrQYqbY
http://senseable.mit.edu/wikicity/rome/
324. Useful Resources
•  Google Cardboard main page
•  https://www.google.com/get/cardboard/
•  Developer Website
•  https://www.google.com/get/cardboard/developers/
•  Building a VR app for Cardboard
•  http://www.sitepoint.com/building-a-google-cardboard-vr-app-in-unity/
•  Creating VR game for Cardboard
•  http://danielborowski.com/posts/create-a-virtual-reality-game-for-
google-cardboard/
•  Moving in VR space
•  http://www.instructables.com/id/Prototyping-Interactive-Environments-
in-Virtual-Re/
325. Resources
•  Excellent book
•  3D User Interfaces: Theory and Practice
•  Doug Bowman, Ernst Kruijff, Joseph, LaViola, Ivan Poupyrev
•  Great Website
•  http://www.uxofvr.com/
•  International 3DUI group
•  Mailing list, annotated bibliography
•  www.3dui.org
•  3DI research at Virginia Tech.
•  research.cs.vt.edu/3di/
329. Steps for Setting up Wikitude
1.  Register for Wikitude account
•  http://www.wikitude.com
2.  Register for web hosting account
•  http://www.000webhost.com
•  Or use own domain and server hosting
350. Customize Additional Features
AR Application
Button
How it works?
•  Additional HTML5, WebVR
•  Allows customization
•  Mobile devices for HMD
(eg. Google Cardboard)
1
2 Additional VR360 on HMD
351. Customize Additional Feature
•  AR Application /AR Browser
•  2D image-based AR tracking
•  HTML5 in-app web browser for additional feature
•  Include a series of VR360 content
•  Requirements
•  AR authoring software /platform
•  Online hosting for additional feature /content
374. Wikitude Studio (cloud-based)
•  Benefits
•  Quick authoring process
•  Instant AR/VR experience generated
•  Generated content can be used in SDK
•  Current limitation
•  Limited control for some multimedia elements
•  Fully rely on network performance
•  SDK may be costly for further implimentation
•  Suitability
•  Exhibition
•  Prototyping
•  Info /edutainment multimedia projects
•  Industrial applications