Keynote presentation given by Mark Billinghurst as part of the OzCHI 2015 Workshop on Science Fiction and the Reality of HCI, December 7th, 2015.

1. MAY THE FORCE BE WITH YOU:
USING FANTASY TO CREATE A BETTER REALITY
Mark Billinghurst
mark.billinghurst@unisa.edu.au
December 7th 2015
HCI and Science Fiction, OzCHI 2015

2. VISION AND MOTIVATION

3. My experience as a boy – seeing Star
Wars for the first time in 1978 inspired
years of PhD research..

4. 1977 – StarWars

7. Augmented Reality
1. Combines Real andVirtual Images
• Both can be seen at the same time
2. Interactive in real-time
• The virtual content can be interacted with
3. Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

8. Making the StarWarsVision Real
• Combining Real andVirtual Images
• Display Technology
• Interacting in Real-Time
• Interaction Metaphors
• Content Registered in 3D
• Tracking Techniques

9. ARTracking (1999)
• ARToolKit - marker based AR tracking
• Nearly 1million downloads, multiple languages
Kato, H., & Billinghurst, M. (1999). Marker tracking and hmd calibration for a video-based
augmented reality conferencing system. In Augmented Reality, 1999.(IWAR'99)
Proceedings. 2nd IEEE and ACM International Workshop on (pp. 85-94).

10. AR Interaction (1999)
• Tangible AR Metaphor
• TUI (Ishii) for input
• AR for display
• Overcomes TUI limitations
• merge task and display space
• provide separate views
• Design physical objects for AR interaction
Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., & Tachibana, K. (2000). Virtual object
manipulation on a table-top AR environment. In Augmented Reality, 2000.(ISAR 2000).
Proceedings. IEEE and ACM International Symposium on (pp. 111-119).

11. Shared Space (1999)
• Face to Face interaction,Tangible AR metaphor
• Easy collaboration, Users as if handling real objects
Billinghurst, M., Poupyrev, I., Kato, H., & May, R. (2000). Mixing realities in shared space: An
augmented reality interface for collaborative computing. In Multimedia and Expo, 2000. ICME
2000. 2000 IEEE International Conference on (Vol. 3, pp. 1641-1644).

12. AR Conferencing
• AR for Remote Collaboration
• Virtual video of remote people
• Provide enhanced spatial cues
• Anchor communication back in real world
• Increased sense of co-presence
Billinghurst, M., & Kato, H. (2000). Out and about—real world teleconferencing. BT
technology journal, 18(1), 80-82.

13. Multi-ViewAR Conferencing
Billinghurst, M., Cheok, A., Prince, S., & Kato, H. (2002). Real world
teleconferencing. Computer Graphics and Applications, IEEE, 22(6), 11-13.

14. Example: 1998 vs. 2008
CPU: 300 Mhz
HDD; 9GB
RAM: 512 mb
Camera: VGA 30fps
Graphics: 500K poly/sec
1998: SGI O2 2008: Nokia N95
CPU: 332 Mhz
HDD; 8GB
RAM: 128 mb
Camera: VGA 30 fps
Graphics: 2m poly/sec

15. Mobile Collaborative AR (2005)
Henrysson, A., Billinghurst, M., & Ollila, M. (2005, October). Face to face collaborative AR on mobile
phones. In Mixed and Augmented Reality, 2005. Proceedings. Fourth IEEE and ACM International
Symposium on (pp. 80-89). IEEE.
• ARTennis
• Shared AR content
• Two user game
• Audio + haptic feedback
• Bluetooth networking

16. Other SciFi Inspired Science
• Dozens of examples
• Communicator
• Star Trek (1964)
• Martin Cooper (1973)
• Taser
• Tom Swift (1911)
• Jack Cover (1974)
• Waldo
• Heinlein (1940s)
• Waldo Manipulator (1950s)

17. The List Goes On..
• Organ Transplant (1818)
• Submarine (1870)
• Credit cards (1887)
• Lasers (1898)
• Helicopters (1890’s)
• Automatic Doors (1899)
• Internet (1904)
• Atomic Energy (1914)
• Taser (1911)
• Humanoid Robot (1927)
• Smart Watch (1946)
• Jet pack (1928)
• Satellites (1945)
• Universal Translator (1945)
• Earbuds (1953)
• GPS (1965)
• Tablets (1968)
• Mobile Phone (1964)
• Voice Control (1968)
• Driverless Car (1990)
• Etc …

18. Observation
• What if..
• People could live forever
• Matter could teleport instantly from place to place
• Robots could have human brains
Science Fiction creates a vision
that can motivate researcher

19. THE RESEARCH GAP

20. Clipper of the Clouds
Jules Verne 1887
Helicopter
Igor Sikorsky 1939

21. “Anything that one man can imagine,
another man can make real.”
Sikorsky

22. Evolution of Science Fiction
• Early authors (1800s – 1920s)
• Mary Shelley, Edgar Allen Poe, Jules Verne, and H.G. Wells
• Educated but not scientists, technology literate
• Magazine Era (1920s – 1960s)
• Pulp Magazines
• In 1953 of 18 leading writers, 8 had degrees in science or engineering
• Hero scientist character
• New Age Science Fiction (1970s-)
• Move to novels, movies
• Authors trained as writers, not scientists

23. There is .. a link between science fact and the ideas
that emerge in science fiction and fantasy.
Science fiction authors are inspired by actual
scientific and technological discoveries, but allow
themselves the freedom to project the possible
future course of these discoveries and their potential
impact on society..
Scientists, in turn, often derive inspiration from the
imaginative possibilities that exist in fictional worlds..
David R. Smith
http://eurekafactory.net/2014/07/21/help-us-obi-wan-why-science-needs-science-fiction/

24. Virtuous Cycle
Author
Scientist
Vision
Technology
Book, Movie
Inspiration

25. Making A Monster
• Step 2 Left as Exercise for Reader
Step 1:
Assemble
Body Parts
Step 2:
Apply
Technology
Step 3:
Reanimated
Monster

26. Observation
• Inspiring innovation, but leaving room for
technical development
• Using today’s technology to create Science
Fiction experiences
Science Fiction leaves the
details of how to achieve the
result up to the reader/viewer.

27. EXPOSING INNOVATION

28. Bringing Interface Concepts to Public
• Science Fiction enables millions of people to learn about
advanced interface concepts
• More people have seen Augmented Reality in movies
than have read academic papers describing it
• CNN AR Conferencing – 2.8 million views
• Billinghurst AR Conferencing papers - < 2,500 citations
• Other Examples
• Teleconferencing – 2001 Space Odyssey (1968)
• Gesture Interaction – Minority Report (2002)

29. 2008 – CNNAR Conferencing
https://www.youtube.com/watch?v=thOxW19vsTg 2,764,000 views

30. 2001 Space Odyssey
https://www.youtube.com/watch?v=vWwo6JpMceg 31,000 Views

31. Minority Report
https://www.youtube.com/watch?v=2l_IUAcvfv8 82,000 Views

32. Observation
• More people watch movies, read books than
technical papers
• Easier to understand the user experience
Science Fiction provides a good
way to exposure advanced user
interface concepts to the public

33. SCI-FI AND INTERACTION DESIGN

34. SciFi for Interaction Design
• Science Fiction to improve Interaction Design
• Interaction Design provides opportunities for story telling
• SciFi techniques can be used to tell better stories
• Use Science Fiction to show research vision
• Method
• Set high level vision
• Explain vision through story, sketches, movies
• From stories identify technology needed
• Based on user feedback, develop working prototype

35. Interaction Design Process

36. Example Storyboard

37. Example Storyboard p.2

38. Google Glass Interface Concept

39. NASA Hololens Concept Demo
• Vision: Work on Mars from your office
• Story and sketches based on vision
• Led to working Demo

40. Chesley Bonestell (1940s)

41. Hololens Story

42. HoloLens Concept Sketches

43. Final NASA HoloLens OnSight Demo
https://www.youtube.com/watch?v=o-GP3Kx6-CE

44. Observation
Science Fiction techniques can
be used to improve the
Interaction Design process
through better story telling

45. Interaction Design to Validate SciFi
• Interaction Design/HCI techniques to validate
Science Fiction interface designs
• Look for relevant CHI, ToCHI papers, etc
• Minority Report gesture interface looks great, but:
• Fatigue after holding arms in air
• Lack of precision for hand movements
• Difficulty for in-arm typing
• Interfaces made for movies, may not work in real life

46. Consider theWhole User

47. Example:SocialAcceptance
• People don’t want to look silly
• Only 12% of 4,600 adults would be willing to wear AR glasses
• 20% of mobile AR browser users experience social issues
• Acceptance more due to Social than Technical issues
• Needs further study (ethnographic, field tests, longitudinal)

48. Example: Google Glass
• Amazing technology, but didn’t consider Social Factors
• Geordi La Forge + Social Acceptance

49. Observation
Interaction Design methods can
be used to validate Science
Fiction technology
• Predict which technology will become widely
used and provide a good experience

50. CONCLUSIONS

51. SciFi Creates a Better Reality by:
1/ Showing a vision of the future
• Allowing people to see what if?
2/ Motivating user interface research
• Researchers left to fill in the gaps
3/ Exposing innovative interfaces to millions
• YouTube views versus citations
4/ Being part of the Interaction Design process
• Storyboarding, concept designs

52. What We Need to Do
1/ Learn how to be story tellers
• Begin with the end in mind
2/ Bring science into Science Fiction
• Show how stories can be made real
3/ Apply Interaction Design guidelines to SciFi
• Focus on the entire user experience

53. www.empathiccomputing.org
@marknb00
mark.billinghurst@unisa.edu.au