Presented at 2017 Information Architecture Summit.
As humans, we are good at engaging different kinds of designs with different kinds of actions. Flexibility is in our nature. Yet, there’s something fundamental about us that makes our experience with a design feel natural, or…distinctly off. This talk draws on ecological psychology to see that natural human behavior is about two things: using information for selecting action, and relying on information for controlling action unfolding over time. Information architecture historically supported selecting, creating an actor-as-conductor of information dynamic. But, IA is increasingly relied on to help control the way action unfolds over time, an actor-as-sculptor of information dynamic. We’ll follow the thread of meaning for both to uncover factors leading to natural vs. unnatural behavior, and what we can do about it. Design examples will come from information environments that vary in how information manifests (from holograms and simulations in mixed reality to smart materials), how the actor engages it (gesture, voice, touch, among other methods), how much agency the system brings (autonomy, to machine learning and shades of intelligence), and how the system manifests to actors (text and visualization).
2. The Tree of LifeArtistic illustration of the tree of life by Evogeneao
I started out in biology; the tree of life was my first taxonomy.
(NOTE: this image is an artistic illustration of evolutionary complexity. For the latest structure with scholarly rigor, see Banfield and Hug: http://
www.nature.com/articles/nmicrobiol201648 )
3. Then I learned that not just biology had a classification system…
9. Earth image: NASA Apollo 17 crew https://upload.wikimedia.org/wikipedia/commons/thumb/7/78/The_Blue_Marble.jpg /1023px-The_Blue_Marble.jpg
Sagan (1994)
And, that’s home. That’s us. (Carl Sagan, Pale Blue Dot, 1994). A bunch of tribal hunter-gatherer poets. As designers, we’ve opened a lot of
wormholes to new information around here, designed information. As humans, we’re really good at assembling our actions with designed information.
Some designs are complicated, hard to understand, and we cope, and some are clear, aligned with out values, and we thrive. But whether we are
engaging something difficult or clear, there’s something deeper about us as humans that makes the way we behave feel natural, or distinctly off.
20. Image: Barcroft Media via Daily Mail, 3D image of human brain connections
That changes the role of these beautiful brains we walk around with. Our brains don’t have to do all this processing, predicting exactly how we’re
going to behave before we do it. Our brains act as coordinators: they coordinate the way we assemble our actions with information in the world
around us.
22. Natural human behavior
Selecting
action
Golanka (2015)
Controlling
action
Natural human behavior is about two things: selecting action and controlling action.
(NOTE: if you read this Golanka 2015 paper, I recommend as background, first reading her paper with Andrew Wilson, “Embodied cognition is
not what you think it is,” available in full from Frontiers in Psychology.)
25. Selecting action
Switching action
Styling action
ACTOR AS CONDUCTOR
Berrypicking model for search: Marcia Bates (1989)
Golanka (2015)
Cascade of selecting, switching, styling
Anticipating-aligning with the performance of information
We’re also conducting when we perform digital search. In Marcia Bate’s berrypicking model for search, the actor anticipates the performance
of the domain indexing in selecting and styling her initial wordings of her query. She then aligns with the meaning of the results, grabbing the
good stuff, and then selecting and styling and switching her query to anticipate further performance, aligning with the meaning of the results,
further refining her anticipation of the performance of the system. Actor-as-conductor is a cascade of selecting, switching, styling in which the
actor is anticipating and aligning with the performance of the system of information.
28. ACTOR AS SCULPTOR
Golanka (2015)
Controlling
action unfolding
continuously
over time
The industrial design of a car supports actor as sculptor. The steering wheel, the gas and brake pedals, and the transparent windshield are
all streaming information that the actor relies on to control action unfolding over time; the actor feels as if she is wielding the dynamics of the
surroundings so that she and the car are receiving the right road, and the lane stays centered around her and the car, and the other cars
flow off to her sides instead of into her.
29. Golanka (2015)
Controlling
action unfolding
continuously
over time
ACTOR AS SCULPTORACTOR AS CONDUCTOR
The industrial design also supports actor-as-conductor of information. A glance at a map to select the right road. A glance at the speedometer to
decide to switch her actions to slow down: she engages actor-as-sculptor behavior, relying on the shifting position of her foot on the brake pedal, and
the way the visual information is streaming through the windshield to control how she slows down continuously over time.
(NOTE: Experienced drivers are really bad at mimicking the act of driving. That’s because drivers don’t decide in their heads how to handle the steering wheel or adjust
the gas and break pedal positions. They rely on the continuous flow of information to help them choreograph the behaviors of slowing down and choosing the right
road and staying in the lane and moving around a car. Without this continuous flow of information to rely on, the meaning of the behavior is broken.)
30. ACTOR AS SCULPTORACTOR AS CONDUCTOR
Historically,
our field
lived here
Historically, our field lived here– actor as conductor– and actor as sculptor was the stuff of industrial design. For actor-as-conductor designs, there is of
course some physical action that has to happen to get the cursor on the pixels, or get the fingertip on the target, or say the words to trigger the
response of the system, but there were lots of degrees of freedom for how the actor controlled physical actions over time. Whether the actor was
sitting down, or walking, or reclining, and the arc and speed of the motions to work a mouse or tap a screen didn’t participate directly in the meaning of
the activity.
34. ACTOR AS SCULPTORACTOR AS CONDUCTOR
Designs in Autodesk Revit and VR by Samuel Arsenault-Brassard
Building architects are experimenting with the handoffs across conducting and sculpting with information. This architect has detailed a building
in architectural software (conducting with information). He brings the building into 3D space in order to sketch some new concepts. He scales
the building down so he can walk around it and look down on it. He relies on information about the way the building occupies space to guide his
hand motions as he sketches new towers into the existing geometry. He can feel how the curves will occupy space with respect to the rest of
the building.
36. ACTOR AS SCULPTORACTOR AS CONDUCTOR
Work with holograms as physical objects
MIT is experimenting with focused sonar that creates mechanical force against an actor’s hand. Now actors can do things like touch holograms,
and explore their surfaces and textures. An actor could work with holograms as physical objects, relying on haptic information to help control
physical actions continuously over time. (Image: MIT)
37. ACTOR AS SCULPTORACTOR AS CONDUCTOR
High-precision motion tracking
collapses degrees of freedom of
spacetime action into meaning
Image: Google Soli project
Google is using radar to very precisely track the position and motion of human fingers. The actor can mimic things like turning a dial, moving a
slider, pushing a button. These are still things for conducting (selecting, switching, styling), but there is a tiny bit of actor-as-sculptor: the actor’s
fingers become tactile information for each other to rely on to control action continuously over time. It’s just fingers now, but soon it will be hand,
arm, the actor’s entire body position at high precision that could participate in sculpting with information.
38. Actor as
Conductor
Selecting action, switching
action, styling action
Dynamics
of
surroundings
Concepts
INFORMATION
Physical
alignment
Semantic
alignment
MEANING
Golanka (2015)
Golanka (2015)
Actor as
Sculptor
Controlling action
unfolding over time
Structure of natural human behavior
Those are just a few examples of how we’re stating to need to worry about both human behaviors. Now I want to build out the structure of
natural human behavior for both of these. For actor as conductor, the meaning is semantic alignment: why we’re selecting an action, why we’re
switching action, why we’re styling the action. For Actor as sculptor, the meaning is physical alignment. For actor as conductor, the information
is about concepts; for actor as sculptor, the information is about the dynamics of the surroundings. Let’s look at what that means.
(NOTE: if you read this Golanka 2015 paper, I recommend as background, first reading her paper with Andrew Wilson, “Embodied cognition is
not what you think it is,” available in full from Frontiers in Psychology.)
39. This is not information for surfing
Underlying dynamics
This is not information for surfing. This is how wind transfers friction to water molecules, pushing and pulling them to form a wave. This is the
underlying dynamics that create a wave, but this is not information for surfing.
40. This is information for surfing
Invariant structure–the shape of the opening–creates affordances
for a surfer to stay inside this place made of liquid information
This is information for surfing. This expert surfer doesn’t see these dynamics, he sees the enclosure of a wave as a place to inhabit. And he aligns
his actions with information about the wave continuously over time to keep the enclosure around him. The shape and slope of the opening of the
wave is information that he can use to position the tip of his surfboard. He is calibrated to the way he must align his actions to this information; the
information becomes affordances for surfing for him. And when the shape of the opening changes, when he feels water hitting his back, he knows
that his place is collapsing and he needs to switch action, engaging other affordances, to quickly get out.
42. This is information for goal keeping (soccer)
Point light image: https://quote.ucsd.edu/cogs91/
This is information for goal keeping for soccer. Twelve points of light are enough to give information to a goal keep about a human kicking a soccer
ball. The goal keep can align her actions to this information to prepare to stop the ball. She doesn’t have to have seen this particular human kicking
a ball, she doesn’t have to have seen ball kicking from this particular perspective in this particular light. She sees the relationships among how one
foot plants while another moves with respect to the knee. This becomes invariant structure, information about ball kicking, that work as affordances
to align with for a goal keeper. (NOTE: visit the URL above to see the animated gif to get the idea of this information.)
44. For actor as conductor, we assemble semantic alignment (why are we selecting, switching, styling an action?) with information about concepts by
way of convention. For actor as sculptor, we assemble physical alignment with information about the dynamics of our surroundings by way of
affordances. We see that conducting with information is not dependent on time. We can conduct in quick succession, or over weeks or lifetimes.
Time does not factor into whether conducting with information feels natural or not. For actor-as-sculptor, natural behavior is directly dependent on
being continuous over time. And that places some very important demands on the kind of information that can support actor as sculptor.
45. For actor as sculptor, information must be dense, persistent, and lawful. And it must be made of perceptual information. Language is not dense
enough, persistent enough, (though it acts as if it’s lawful across an actor’s lifespan) to rely on to control action continuously over time. The two
key things actors use perceptual information for to sculpt with information are: orienting & wayfinding, and prospecting with objects. We’ll look at
both of these.
46. Orienting & Wayfinding
Dense, persistent, lawful
Visual Information
Surfaces
Edges
Textures
Proprioception, Kinesthesia
We are always orienting ourselves physically in our surroundings. We humans are wired for proprioception and kinesthesia: we know where
our bodies are in space, and we know when we’re moving. We also rely on visual information for information about the relationships among
surfaces, edges, and textures in our surroundings that we can use as affordances for us to move: this path is walkable to me, these stairs are
climbable to me, this gravelly hill is not passable to me. And this information is dense, persistent, and lawful. We can rely on it to control our
actions continuously over time.
47. Wayfinding
Haptic Information
Dense (enough)
persistent (enough)
lawful
Simulates surfaces, edges
This woman has low vision and can’t rely on visual information for wayfinding. She’s wearing a device that simulates information about
obstacles as haptic information. The haptic information is much less dense than visual information, and is not persistent (it’s not a constant
flow, but kicks in with proximity to obstacles). But it’s dense enough, persistent enough, and lawfully specifies obstacles in her surroundings
that she can use as information for orienting and wayfinding.
48. Wayfinding
Sonic Information
Dense (enough)
persistent (enough)
lawful
Incoming cars
Train/no train
Crossing light timers
Conversations
Physical obstacles
Incoherent with “everyday sounds” for wayfinding
Gaver (1994)
UNNATURAL BEHAVIOR
This system can use sound instead of haptic information about obstacles. Sound is dense enough, persistent enough, and lawful, but the problem with
sounds is that it’s directional, but not spatial. It piles up. That’s great for music, but not for wayfinding. Haptic information can use spatial relationships to
represent multiple obstacles and their relative positions. Sounds just pile up. The other problem with sound fir wayfinding is that humans rely on many
other very important sounds: incoming cars, train/no train, crossing light timers, conversations. And the sounds about physical obstacles steps on those so
they are not dense, persistent, and lawful. This is unnatural behavior.
(NOTE: see Gaver 1994 in references for an excellent discussion of ecological taxonomy of “everyday sounds” humans rely on for information.)
49. So we see that we rely on many kinds of perceptual information for orienting and wayfinding, and we see that, in our designs to support this, we must
ensure the information we use is not only dense enough, persistent enough, and lawful, but coherent with the other perceptual information actors are
relying on for the activity. So, we’ll add coherence as a design consideration.
50. VR has a coherence problem
"I started forgetting which walls
were virtual and which ones were
real, and I ended up walking into a
few very solid walls"
“A player attempted to push
his hand through the floor to
sculpt something’”
Disrupts critical information for
placetime orienting and wayfinding
Dense (enough)
persistent (enough)
lawful
UNNATURAL BEHAVIOR
Virtual Reality also has a coherence problem. VR disrupts critical information for placetime orienting and wayfinding. Players of VR games report things like, “A
player attempted to push his hand through the floor to sculpt something.” Another: “I started forgetting which walls were virtual and which ones were real, and I
ended up walking into a few very solid walls.” These designs are mitigating this by adding in wall alerts and visual grids to show were physical walls are. These
things are dense enough, persistent enough, and lawful to represent the walls. The problem is that they are like aroras from another reality slicing through the
information from the present reality. They disrupt the dense, persistent, lawful information in the present reality for orienting and wayfinding and ask the actor to
align with two sets of conflicting information. That’s unnatural behavior. Quotes from: http://www.businessinsider.com/virtual-reality-gamers-are-literally-colliding-with-the-physical-world-2016-4
54. Prospecting with objects
Image: https://quote.ucsd.edu/cogs91/
Color relationships become affordances for banana edibility
Surfaces
green : yellow : brown
We don’t just start using objects, we prospect with them using information about them. If we have experience eating bananas, the relative
amounts of green:yellow:brown color on the surface of the peel becomes information to us. We become calibrated to the relative colors to
know that a banana like this will have a peel that droops, we anticipate seeing brown spots on the fruit, we’d expect the fruit to be a little mushy
and sugary. This information becomes affordances that we can rely on about whether this banana is edible to us.
object affordances: Gibson (1979)
56. Are our simulated objects explorable?
Do they know what kinds of objects they are?
Prospecting simulated objects
What kinds of information
should they simulate to reveal
new underlying dynamics?
New kinds of transformations
Permutational Computational
Smart Materials Same concern as simulated objects
Tactile/HapticVisual Sonic Same information as physical objects
What about prospecting simulated objects? We can simulate the same kinds of infomraitno that physical objects have: visual, tactile, sonic. But we can also
infuse simulated objects with new kinds of underlying dynamics that physical objects don’t have: new kinds of transformations. But, we also need to infuse
them with information about these new kinds of dynamics that humans can rely on. We need to start thinking about what a permutational affordance is like;
what a computational affordance is like. Smart materials have the same concern as simulated objects: smart materials are physical materials that are
infused with new underlying dynamics. How do we coax smart materials to radiate information about their underlying dynamics for humans to calibrate with
and use as affordances? (NOTE: We also need to ask, do we truly need a spacetime simulation? If the information is more abstract than the way the object orients and embeds in spacetime
and relative to the actors proportions, are we just making a 3D hologram versio of a skeuomorphism?)
57. Simulations have no mass!
Prospecting simulated objects
“There's no physical furniture
to break your fall when you
try to sit in a virtual chair.
Someone asked, Where did
that bruise come from?
I achieved presence.”
UNNATURAL BEHAVIOR “Breaking the ecological law”
Wilson (2012)
Yet…
they provide affordances for physicality COHERENCE
Tactile/HapticVisual Sonic
touch-ability, throw-ability, transformability
sit-on-ability, climb-on-ability
Yet…
one physicality affordance is not lawful
Visual
Explorability leads to a coherence problem for simulated objects. Simulations have no mass! Yet, they provide affordances for physicality, we perceive them as objects that
take up spacetime embedded in our surroundings. Get dense, persistent, lawful visual and haptic and tactile and sonic information, depending on how they’re wired up, about
touch-ability (surfaces and textures), pickup-ability, throw-ability, transformability, and so on as far as our imaginations take us. Yet, one aspect of their affordances for
physicality is not lawful. We get visual information for things like sit-on-ability, climb-on-ability. Surfaces and textures suggest the lawful presence of mass. These are
affordances we’ve been attuned to our whole lives, and we will engage with. One person in VR recently said: “There’s no furniture to break your fall when you sit on a virtual
chair. Someone asked, ‘Where did that bruise come from?’ I achieved presence.” Andrew Wilson, a present day Ecological Psychologist, would tell us we are breaking the
ecological law by simulating affordances for properties that our objects do not lawfully provide. That’s unnatural behavior.
59. So, that’s the structure of actor as sculptor behavior. Now let’s build out actor as conductor. Information to support conducting can take the form of both
language and perceptual information. Language for conducting is about coordinating and understanding. We’ll start with coordinating.
60. Ecological psychology tells us that natural human conversation is about coordinating. When someone speaks and someone listens, that evokes concepts.
Concepts point to meaning. Meaning points to values. And that’s enough to evoke a virtual array of cultural possibilities among the participants. Something as
seemingly here-and-now as asking which movie should we see tonight locates speakers and listeners in place and time, and evokes all of their placetime histories
as an array of cultural possibilities. Speaking is anticipating cultural possibilities by improvising wordings. Listening is aligning with the discovered nature of the
cultural possibilities, and entraining with rhythms, timing, pronunciation, grammar of the participants in the conversation. A conversation is a creative tension:
anticipating and aligning; improvising and entraining. The acts of conversation feed back into the structure of the cultural possibilities, changing the structure over
time. The overall behavior of this system of conversation is about coordinating actions among the participants to realize values.
64. So, we see that natural human conversation is about coordinating. Now let’s look at using language for understanding.
(NOTE: I am using “understanding” as the label here to mean the notion of design for understanding in Morville, Rosenfeld, Arango (2015).)
65. When an actor engages a design that supports something like shopping, banking, creating, hailing/styling a service, etc., the actor is engaging with language based on classification: the underlying
information architecture of the designed system. And the purpose of the behavior is understanding/performing for a good continuation. The actor engages the design and that evokes concepts.
Concepts point to meaning. Meaning points to values. And that’s enough to evoke a virtual array of classificational possibilities. This takes the form of the underlying system of information architecture.
The actor anticipates the performance of the classification by improvising selecting/switching/styling. Selecting/switching/styling can take the form of browsing, searching, commanding, hailing,
parameterizing, etc., whatever the design facilitates. The actor then aligns with the meaning of the performance of the system (discovered nature of the classificational possibilities). And entrains with
the labeling and grammar of the system. Entraining is vital here: if the actor does not entrain to the labeling and grammar of the system, the virtual array of meaning falls apart. We know from our field
of IA that a system must be clear, and resilient across channels and contexts. The behavior of the system is about understanding/performing for a good continuation.
66. Now we are playing with the parts of this system: autonomous digital agents offload from the actor the tension of anticipating/aligning with the classificational possibilities. The
agent does this work. But, when the agent needs to interrupt the human with a decision point or status, the agent must bring the right kind of information to let the human
anticipate/align with the digital agent for a good continuation. We are also injecting machine learning into the system. That can greatly expand what is allowable for entraining
with the system. Entraining is still vital, but the human has more flexibility in what is allowed for labeling and grammar. Machine learning in which the system is expanding the
nature of the underlying classificational possibilities will expand the scope of what the human needs to anticipate and align with. I think that things like machine learning and
conversational UI are really about expanding the tolerance for imprecision the human must have to engage the system.
67. Human-System imprecision tolerances
NARROW Tolerance
Precise entrainment
WIDE Tolerance
Forgiving entrainment
Natural Language Processing
Machine Learning
(entraining actor’s wordings)
Conversational UI
Standard searching, voice commanding
Gestural UI
(degrees of freedom of motion, subtle vs elaborate)
For any given system, the system has a tolerance for imprecision in what is needed to engage it. A narrow tolerance for imprecision would be a system that requires very
precise entrainment with labeling and grammar. A wide tolerance for imprecision would be a system that has forgiving entrainment with labeling and grammar. Something
like standard searching and voice commanding have narrow tolerance for imprecision. We can do things like autosuggest, and spelling forgiveness, but these are generally
very precise labeling and grammar requirements. Conversational UI expands the tolerance for imprecision by suggesting the classificational possibilities ahead; this gives the
actor a hint of the performance of the system as she is selecing/switching/styling. Natural language processing expands the tolerance for imprecision. Machine learning in
which the system entrains on the actor’s particular wordings greatly expands the tolerance for imprecision with a very forgining entraining requirement on the human.
Something like gestural UI right now has a narrow tolerance for imprecision. I believe the tolerance for imprecision should be much more forgiving to mime concepts, both in
terms of the degrees of freedom of motion to give the system the gist of the concept, and in whether the human can use subtle motions or must do elaborate gestures.
68. So, we’ll add imprecision tolerance as a design consideration. For actor as conductor, we can also use perceptual information. This takes the
form of mapping: infusing concepts with perceptual qualities in the form of visualizations and notifications: maybe a sound or haptic buzz or color
change to signal a state change.
69. Mapping
Air flow and heat transfer simulations, Autodesk
Visualization Values-realizing, good continuation, clear, resilient
Curvature mapped to airflowColor mapped to temperature
This is a visualization of the air flow and heat transfer in a hospital operating room on the left, and an airport walkway on the right. Color is
mapped to different temperature ranges. The arc of the lines shows the direction of the airflow. We could describe something like this in words,
but this visualization captures the ambient comfort of these places as a quick, visceral understanding. That’s because perceptual information
and language feel different in terms of the ease of the flow of engaging them.
70. Perception flows easily
Tacit
Reflexive
photo credit: Jason Pratt via Creative Commons
Language is viscous,
more work to flow
Awareness
Associativity
Requires attention
These kinds of information have different viscosities, or ease of flow.
72. So, we’ll add viscosity as a design concern. We see that the structure of natural human behavior for actor as sculptor is physical alignment using the
dynamics of the surroundings by way of affordances. It relies on perceptual information for orienting & wayfinding, and prospecting with objects. Perceptual
information for sculpting can take many forms, but must always be coherent with the way other kinds of information are being used for the activity.
Information for sculpting must be dense enough, persistent enough, and always lawful. Natural human behavior for actor as conductor is semantic
alignment using concepts by way of convention. Both language and perceptual information can be used to support conducting. Language for coordinating
takes the form of human conversation. Language for understanding takes the form of systems of classification. Perceptual information for conducting is a
mapping to concepts using visualization or notification. Information to support conducting must be values-realizing, a good continuation, clear, and resilient.
76. Image: DMITRIS_K Big Think
Disrupting the ecology of information
If our designs feel deep-down unnatural, and we can’t pin it on usability or usefulness or lack of clarity, or any of our other
ways to say what’s good, maybe it’s because we’re disrupting the ecology of the ways our human actors are relying on
information to behave.
77. Human behavior is events aligning,
Tuned to align our actions
For 100s of 1000s of years, and more across the branches of the tree of life, our perceptual systems have been tuned to
align our actions…