This document discusses natural user interfaces (NUI) and provides an overview of NUI technologies including the Kinect, Wiimote, and Surface. It describes how these devices track motion and gestures using technologies like infrared lasers and cameras. Examples are given of NUI uses in media, operating systems, and science. Tools for developing with these devices like the Kinect SDK are also mentioned. The document concludes with the direction of the author's own work building an API and gesture language to simplify NUI development.
3. agenda
• Introduction
• What is NUI/NUX
• NUX In the media and the wild
• Real-world devices
• Tools, SDK’s, …
• Where is this going?
• Demo using Kinect and Wiimote
4. about mike
• Principal global technologist, Advanced Technology
Services, SunGard Global Services
• Technology focus on .NET, cloud, parallel systems, rich
interfaces and NUX
• mike@heydt.org, michael.heydt@sungard.com
• @mikeheydt
• 42spikes.com
5. 55
Advanced Technology
What is Advanced Technology?
SunGard Consulting Services Approach
Typical Client Engagements
Our consultants offer deep domain experience and a broad set of leading edge technology
skills, allowing companies to optimize their business processes
We use a variety of Agile project management and execution techniques to facilitate quick
delivery and reduce risk
We take an objective approach in the selection of tools and products, and are comfortable
working with a variety of vendor products and platforms
We have applied our advanced technology skills to provide solutions to customers in finance,
energy and media sectors
Typical assignments involve deploying highly skilled specialists with relevant domain
knowledge who work with customers in a collaborative way to design and build custom
solutions
MapReduce/Hadoop
Complex Events Processing
GPU Mathematical Pricing
Mashups
Web 2.0
Cloud & Grid Computing
Functional Languages
Mobile Applications
RIA (Silverlight/Flex)
Semantic Web
REST/JSON
Grails
6.
7. opening questions…
• Who here has used a Kinect on the Xbox?
• Who has programmed a Kinect?
• Who has used a WiiMote on the Wii?
• Who has programmed a wiimote in their application?
• Who has seen Minority Report?
8. what is nui/x?
• In computing, a natural user
interface, or NUI, is the common
parlance used by designers and
developers of computer interfaces
to refer to a user interface that is
effectively invisible, or becomes
invisible with successive learned
interactions, to its users.
• NUI Covers the class of interfaces
where the user is enabled to
interact with the computer in a
more natural manner.
9. genesis
• Conceptually has been thought of for several decades
• Made popular by Minority Report
• iPad has changed the game with single-task gestural
apps
• Xbox Kinect has broken open the flood gates for NUX
• NUX is starting to show adoption in operating systems
through the use of tablet and trackpad gestures
10. in the media and the wild
• Xbox
• Operating systems
• Movies and television
• Science
26. kinect
• Laser Depth Imager
• Video Camera
• Microphone Array
• Very XBox Specific Interface (non-standard USB)
• Microsoft SDK, non-commercial license
• $150
32. elements of nux
• Skeleton Tracking
• Gesture (Body, fingers) Identification
• Voice Recognition
33. the how of the kinect
• Perform initialization gesture (helps, but not always
necessary)
• You are scanned and tracked by an IR laser
• Depths are measured to the millimeter at 320 x 200 +
• From depths skeletons are identified and tracked
44. Thoughts on the Wiimote
• An interesting device, but…
• It’s first gen
• Looked good prior to the Kinect
• All capabilities can be duplicated with Kinect (point
tracking) and a cell phone (accelerometer, gyro)
• And a cell phone can provide additional uses such as
robust data entry
45. speech
• Kinect has a very good set of microphones
• Speech recognition is not in the kinect - use the .NET
libraries
• Must be bridged with Microsoft Speech Platform SDK
• I can’t stress this enough – this is excellent
augmentation
46. gesture recognition
• Not built into any of the
devices
• Typically implemented with
Dynamic Time Warping
algorithms (DTW)
• Record sequences of skeleton
locations, match against live
streams
47. difficulties
• Many devices
• Immature libraries
• Small body of work
• Many API’s, even for each
individual device
• Kinect is non-standard
USB
• All devices are separate
and don’t interoperate
• All work on a single
system – No support for
moving physical location
• No standard pattern
language for NUX
• No “canned” gesture
engines
48. what i’m building
• NuiDotNet (nuidotnet.codeplex.com, nuidot.net)
• Abstracts devices from the application
• Allows configuration and location based construction
of services
• Multiple and prioritized services and interactions
mapped to application functions
• Gesture engine and domain specific language for
gestures
49. directions
• Define useful set of gesture patterns for data
manipulation
• Build finger / hand tracking capabilities
• Make the DTW gesture engine robust
• Tracking of specific parts of the body
• Architecture and software to plug into business
applications
• Provide a library of pluggable UI elements and gestures
50. uses for business
• Interactive data manipulation and visualization
• Free-form visual application development
• Media presentations
• Home control
• Many still to be determined
51. demo and the tools
• Hardware: Kinect, WiiMote
• Libraries: Kinect SDK, Wiimotelib
• Language and GUI: C# / WPF
52. resources
• ^ Buxton, Bill (2010-01-06). "CES 2010: NUI with Bill Buxton". Microsoft Research.
• http://www.microsoft.com/surface/en/us/default.aspx
• http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/
• http://www.youtube.com/watch?v=1-tMp4WkQjA
• http://openni.org/
• http://silverlightkinect.codeplex.com/
• http://mono-mig.sourceforge.net/
• https://42spikes.com