A presentation Julian Bleecker and myself gave at SXSW2012 about the mind and consciousness as an interface.

1. The Mind and Consciousness
As an Interface
Julian Bleecker & Nicolas Nova
SXSW12 | Austin, TX
After Julian presented the cultural backdrop for mind and consciousness interfaces, I want to focus on the here
and now, explain you the technologies and discuss the interaction design opportunities and limits

2. Adam Wilson’s EEG-generated tweet

3. Cortical control of a prosthetic arm for self-feeding by Meel Velliste, Sagi Perel, M. Chance
Spalding, Andrew S. Whitford & Andrew B. Schwartz http://www.youtube.com/watch?
v=gnWSah4RD2E

4. Neurowear - necomini: http://neurowear.com/
“Necomimi is the new communication toolthat augments the human bodies and abilities.This cat’s ear
shaped machine utilizes brainwavesand expresses your emotional state before you start talking”

5. Are these interfaces real?
Yes, they all are

6. BCI: Brain-Computer Interfaces
“create a direct communication pathway between a human brain
and any external devices like computers”
“BCI are system that provide direct translation of the activity of
the human brain into command which can control device in real
time.”
Research started with disabled people... now it expands to any kind of users ( as usual in the
history of innovation... “augmentation” )

7. “direct interaction”
A common trope in human-object interactions: lever / mouse / keyboard / touch / voice /
physiological

8. Process > invasive technology
There are two ways to do it
1. Invasive technologies: sensors directly implanted in the brain:
+ = high temporal and spatial definition
- = require surgery, cannot be moved, cover only limited portion of the brain
Picture source: Nick Ramsey (http://www.nick-ramsey.eu/pics/surgery.jpg)

9. Process > non-invasive > EEG
This is the other possibility, no surgery. For the signal acquisition there are many different measurement
methods such as electroencephalography (EEG), magnetoencephalography (MEG), functional magnetic
resonance imaging (fMRI) or near-infrared spectroscopy (NIRS).

10. Process > non-invasive > fMRI
This is what you will need for functional magnetic resonance imaging (fMRI)
fMRI measures changes in blood flow to various areas of the brain over time. It is believed that blood flows to
brain tissue that is active. So, the reasoning goes, if we put someone in the scanner, and ask them to think about,
say, playing tennis, then blood should flow to the part of the brain that is involved with thinking about tennis
playing.

11. Process > non-invasive > EEG
EEG is the most common., electrodes placed on the scalp measure the weak electrical potentials generated by
the brain activity
+: no surgery
-= noise, low spatial resolution (2-3cm 7-11 inches accuracy, which corresponds to the location of areas of
specific brain activities: moving your hands versus listening to music)
Most of the research is in there: data acquisition
Credit photo: http://www.bsp.brain.riken.jp/Photos/EEG%20Acquisition%20-%20%2064-QuickCap
%20NeuroScan%20+%20256-Geodesic%20Net%20EGI%20-%20LABSP,%20BSI,%20Riken.jpg?
w=300&h=300

12. Process > non-invasive > “dry cap”

13. Process > making sense of this
1. Detect specific mental state or activity in certain areas of the brain (you can then say the user is
reading but it’s almost impossible to tell what)
2. Train users to create certain brain activity and use it instead of a motor movement (moving the
hand)

14. Process > making sense of this
delta wave: slow
waves, sleep
theta wave:
drowsiness or arousal
alpha wave: relaxed,
reflecting
beta wave: alter,
working, anxious
thinking,
concentration

15. Process > making sense of this
It was to difficult for us to resist presenting you this domain AND NOT USING the graphic
employed by researchers

16. Process > making sense of this
It was to difficult for us to resist presenting you this domain AND NOT USING the graphic
employed by researchers

17. Process > making sense of this
It was to difficult for us to resist presenting you this domain AND NOT USING the graphic
employed by researchers

18. Typology of applications
Various domains: gaming, spelling applications,
2D cursor control, relaxation tool, game, access
to consciousness/dreams, serious games/ brain
training programs (arithmetic exercises, number
or letter recognition, mental card games), brain
to brain communication, mind-controlled
whatever, zen-like interfaces

19. Typology > touch-free interface
Mind-controlled parachute (Sky1 HD TV show)

20. Typology > touch-free interface for
hostile environment
Mind-controlled parachute (Sky1 HD TV show)
Action at distance: http://sky1.sky.com/gadget-geeks/mind-controlled-parachuteurinal-video-game

21. Typology > thought pattern recognition
Mental state identification to deliver dynamically adapted
content/response
Adaptive Brain-Computer Interface by Audrey Girouard (Tufts University: “Difficulty levels during video game play.
Distinguishing difficulty levels could prove to be an interesting input signal, on which to adapt the interface.The
experiment presented the user with two levels of difficulty of an arcade game (Pacman). Data from nine
participants shows we can discriminate well between the subject playing or resting (94% accuracy,with chance at
50%), as well as discriminate between two difficulty levels and rest periods (77% accuracy,with chance at 33%),
which shows potential for use in an adaptive interface. I investigated the data using both statistical analysis and
machine learning classification”

22. Typology > modern day lie detector
Neuroscientist Daniel Langleben. This image shows average brain regions for 22 subjects during testing. Blue
areas represent brain regions more active when telling the truth, red areas, when lying.

23. Typology > neurofeedback
Practice generating the state of mind most beneficial to
sport/activity
PLX Wave - Xwave headset: http://www.plxwave.com/
By safely detecting your brain's rhythm through a small sensor contact gently placed on the skin of
your forehead, XWave will let you be able to control and float objects in video games by simply
thinking about it, or train your mind to focus and relax on command.

24. Interaction design repertoire
Explicit versus implicit user interactions (or user-driven versus
stimulus-driven interactions)
> “Vocabulary of interaction”: control, detect
Synchronous versus asynchronous (as usual most of the
applications focus on the real-time).
Detection of certain cognitive states / brain activity
Stand-alone brain-computer interface or BCI+other
physiological data (hearbeat, turning one’s head...)
Design parameters we can play with

25. Interaction design repertoire
Language-related
cognitive states
Implicit Explicit
Motor-related
cognitive states
Design parameters we can play with

26. 1. Cognitive state versus semantic
Finding general cognitive processes is not mind reading! Researchers can tell if a user is
processing language but they we cannot easily determine the semantics of the content. There
are projects that trying doing so but it’s really far-fetched into the future (big material, lot of
noise).

27. 1. Cognitive state versus semantic
Presented clip Clip reconstructed from
brain activity
An exception though: movie clip reconstruction using fMRI: The left clip is a segment of a Hollywood movie trailer
that the subject viewed while in the magnet. The right clip shows the reconstruction of this segment from brain
activity measured using fMRI.
Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies by Shinji Nishimoto, An
T. Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, Jack L. Gallan (UC Berkeley): http://dx.doi.org/10.1016/
j.cub.2011.08.031

28. but this is what you will need for this (not to mention the algorithm)

29. 2. Training users
Users have to learn to intentionally manipulate their brain signals, which is not trivial.
Two options 1. users are given specific cognitive tasks such as motor imagery to generate measurable
brain activity, 2. Operant conditioning, provides users with continuous feedback as they try to control
the interface.

30. 2. Training users
Main problem = even with feedback it’s hard to see what you can do with your mind to
control the attention interface on the upper left-hand corner!

31. 3. Signal versus noise: seamful design?
Noise = background noise + lots of data (some from the required brain activity + some from other
activities like moving your head/hand) + as people grow tired. But we can maybe adopt a seamful
design approach.

32. 4. Taking design context into account
Where are you going to use these BCI helmet? Most of the pictures about these projects show
lab pics... and we know that design is about context and taking into account contextual
parameters. (This is the “board of imagination”, a mind-controlled skateboard by Chaotic
Moon Lab)
http://www.chaoticmoon.com/labs/chaotic-moon-labs-board-of-imagination/

33. 5. Context and devices
Specific
type of app
Specific
type of app
Specific
type of app
Different paths... depending on context, you can find about different activity

34. Conclusion
> The “direct translation” trope and how to go
beyond it
> We need to build an interaction design perspective,
ask design issues, not only address technological
problems
> What’s the equivalent of the blue screen of death
for BCI? What happens for social norms in the long-
run?

35. Thanks!
julian@nearfuturelaboratory.com
nicolas@nearfuturelaboratory.com