Presented at 2017 Information Architecture Summit. As humans, we are good at engaging different kinds of designs with different kinds of actions. Flexibility is in our nature. Yet, there’s something fundamental about us that makes our experience with a design feel natural, or…distinctly off. This talk draws on ecological psychology to see that natural human behavior is about two things: using information for selecting action, and relying on information for controlling action unfolding over time. Information architecture historically supported selecting, creating an actor-as-conductor of information dynamic. But, IA is increasingly relied on to help control the way action unfolds over time, an actor-as-sculptor of information dynamic. We’ll follow the thread of meaning for both to uncover factors leading to natural vs. unnatural behavior, and what we can do about it. Design examples will come from information environments that vary in how information manifests (from holograms and simulations in mixed reality to smart materials), how the actor engages it (gesture, voice, touch, among other methods), how much agency the system brings (autonomy, to machine learning and shades of intelligence), and how the system manifests to actors (text and visualization).

1. Marsha Haverty
Acting naturally
with information
@mjane_h
2017 Information Architecture Summit

2. The Tree of LifeArtistic illustration of the tree of life by Evogeneao
I started out in biology; the tree of life was my first taxonomy.
(NOTE: this image is an artistic illustration of evolutionary complexity. For the latest structure with scholarly rigor, see Banfield and Hug: http://
www.nature.com/articles/nmicrobiol201648 )

3. Then I learned that not just biology had a classification system…

4. ...Any subject could.

5. Instance
Instance
Instance
Instance
Instance
Instance Instance
Instance
Instance Instance
Instance
Instance
InstanceInstance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance Instance
Instance
Instance
Instance
Instance
Instance
Instance
And you could structure the information.

6. Instance
Instance
Instance
InstanceInstance
Instance
Instance
Instance Instance
Instance
Instance Instance
Instance
Instance
Instance
InstanceInstance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance Instance
And the whole thing could move with an actor's meaningful action.

7. Instance
Instance
Instance
InstanceInstance
Instance
Instance
Instance Instance
Instance
Instance Instance
Instance
Instance
Instance
InstanceInstance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance Instance
Structure could even facilitate it.

8. Instance
Instance
Instance
InstanceInstance
Instance
Instance
Instance Instance
Instance
Instance Instance
Instance
Instance
Instance
InstanceInstance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance Instance
Information
Architecture
I'll always love biology. But I'd found information architecture. That's home.

9. Earth image: NASA Apollo 17 crew https://upload.wikimedia.org/wikipedia/commons/thumb/7/78/The_Blue_Marble.jpg /1023px-The_Blue_Marble.jpg
Sagan (1994)
And, that’s home. That’s us. (Carl Sagan, Pale Blue Dot, 1994). A bunch of tribal hunter-gatherer poets. As designers, we’ve opened a lot of
wormholes to new information around here, designed information. As humans, we’re really good at assembling our actions with designed information.
Some designs are complicated, hard to understand, and we cope, and some are clear, aligned with out values, and we thrive. But whether we are
engaging something difficult or clear, there’s something deeper about us as humans that makes the way we behave feel natural, or distinctly off.

10. Earth image: NASA Apollo 17 crew https://upload.wikimedia.org/wikipedia/commons/thumb/7/78/The_Blue_Marble.jpg /1023px-The_Blue_Marble.jpg
Girl with Amazon Echo: Android Central, Infraworks AR
Sandbox: Autodesk, Tilt Brush VR: Google, Woman hailing
drone: Mercedez-Benz, self-assembling material: Adventurine,
generative design for a chair: David Benjamin, Autodesk
Variety in designed information
Manifestation
Agency
Action to engage
The information we’re experimenting with around here has more variety in how it manifests to actors, the kinds of actions actors must use to engage it,
and in its agency (autonomy, shades of intelligence).

11. Earth image: NASA Apollo 17 crew https://upload.wikimedia.org/wikipedia/commons/thumb/7/78/The_Blue_Marble.jpg /1023px-The_Blue_Marble.jpg
Natural vs. Unnatural
Information physically present vs. simulated
Actions to engage new vs. familiar
Does not depend on:
Some of these things feel pretty natural, and some of them feel like novelties that we don’t really want to be doing for very long. But the distinction
doesn’t depend on whether the information is physically present or simulated; it doesn’t depend on whether the kinds of actions we use to engage is
are new or familiar. Throwing a hologram across the room or talking at an appliance or hailing a drone can all come to feel perfectly natural. The
distinction between natural and unnatural comes in something deeper still about human behavior.

12. The Observer from Fringe: Michael Cerveres
The nature of human behavior?
We don’t have a handy gadget like the one the Observer on the science fiction show Fringe had through which to peer at the nature of human behavior.

13. The Observer from Fringe: Michael Cerveres
ECOLOGICAL PSYCHOLOGY
The nature of human behavior?
But we do have a very powerful lens called Ecological psychology. This field was founded in the 1970s and is just now coming into its own.

14. Golanka (2015)
ECOLOGICAL PSYCHOLOGY
Events of our actions Events of information
Human behavior is about aligning events
(NOTE: if you read this Golanka 2015 paper, I recommend as background, first reading her paper with Andrew Wilson, “Embodied cognition is not
what you think it is,” available in full from Frontiers in Psychology.)

15. Golanka (2015)
ECOLOGICAL PSYCHOLOGY
Events of our actions Events of information
Human behavior is about aligning events

16. Golanka (2015)
ECOLOGICAL PSYCHOLOGY
Human behavior is about aligning events
Events of our actions Events of information

17. Golanka (2015)
Human behavior is about aligning events
ECOLOGICAL PSYCHOLOGY
Events of our actions aligned with events of information
assemble a system with dynamics that complete a task
Behavior is a system. Behavior includes us, but isn’t only us. It’s the events of information too.

18. We’re aligning our actions with information,
but it feels to us as if we’re wielding the dynamics
of the world to act in task-fulfilling ways
Barrett (2011)

19. We’re aligning our actions with information,
but it feels to us as if we’re wielding the dynamics
of the world to act in task-fulfilling ways
Barrett (2011)
We’re not acting to impose our will on the world,
we’re acting to receive the world in a useful way

20. Image: Barcroft Media via Daily Mail, 3D image of human brain connections
That changes the role of these beautiful brains we walk around with. Our brains don’t have to do all this processing, predicting exactly how we’re
going to behave before we do it. Our brains act as coordinators: they coordinate the way we assemble our actions with information in the world
around us.

21. Natural human behavior?
But, what makes behavior feel natural?

22. Natural human behavior
Selecting
action
Golanka (2015)
Controlling
action
Natural human behavior is about two things: selecting action and controlling action.
(NOTE: if you read this Golanka 2015 paper, I recommend as background, first reading her paper with Andrew Wilson, “Embodied cognition is
not what you think it is,” available in full from Frontiers in Psychology.)

23. Selecting action
Switching action
Styling action
Selecting
Golanka (2015)
Gif of Alondra de la Narra: http://indieclassical.tumblr.com/post/138392136778/exercisesinhumiliation-alondra-de- la-parra-is
Selecting is about deciding for a given task which of the possible actions will we engage (if we’re thirsty and we see a glass of water we decide
to drink from the glass), we decide when to switch actions (as we are reaching for the red glass, someone says, “take the blue glass” so we
switch to reach for the blue glass), and we style the way the action unfolds (if we notice that the glass is fragile, we slow our actions and pick it
up carefully). Animated gif: http://indieclassical.tumblr.com/post/138392136778/exercisesinhumiliation-alondra-de-la-parra-is

24. Selecting action
Switching action
Styling action
Selecting
Golanka (2015)
ACTOR AS CONDUCTOR
Gif of Alondra de la Narra: http://indieclassical.tumblr.com/post/138392136778/exercisesinhumiliation-alondra-de- la-parra-is
When an actor does this with designed information, it feels to her like she’s conducting with information. Browsing a website, the actor
anticipates the performance of the categories as she selects one; then aligns with the meaning of the results to anticipate and selecting more,
sometimes switching to go in another direction. It’s this cascade of selecting and switching that feels like she’s conducting with information.

25. Selecting action
Switching action
Styling action
ACTOR AS CONDUCTOR
Berrypicking model for search: Marcia Bates (1989)
Golanka (2015)
Cascade of selecting, switching, styling
Anticipating-aligning with the performance of information
We’re also conducting when we perform digital search. In Marcia Bate’s berrypicking model for search, the actor anticipates the performance
of the domain indexing in selecting and styling her initial wordings of her query. She then aligns with the meaning of the results, grabbing the
good stuff, and then selecting and styling and switching her query to anticipate further performance, aligning with the meaning of the results,
further refining her anticipation of the performance of the system. Actor-as-conductor is a cascade of selecting, switching, styling in which the
actor is anticipating and aligning with the performance of the system of information.

26. Selecting
action
Controlling
action
Natural human behavior
Golanka (2015)
Selecting is behavior: the action of selecting action
Selecting is an event: the action of selecting action. Now, we’ll look at controlling action.
(NOTE: if you read this Golanka 2015 paper, I recommend as background, first reading her paper with Andrew Wilson, “Embodied cognition
is not what you think it is,” available in full from Frontiers in Psychology.)

27. Image by permission Cherrico Pottery LLC http://www.cherricopottery.com/
Controlling
Controlling
action unfolding
continuously
over time
Golanka (2015)
ACTOR AS SCULPTOR
We are always orienting ourselves physically in place and time with our surroundings. When we act in placetime, we control the way our
physical actions unfold continuously over time with information in our surroundings. When an actor engages designed information that does
this, it feels to the actor like she’s sculpting with information.

28. ACTOR AS SCULPTOR
Golanka (2015)
Controlling
action unfolding
continuously
over time
The industrial design of a car supports actor as sculptor. The steering wheel, the gas and brake pedals, and the transparent windshield are
all streaming information that the actor relies on to control action unfolding over time; the actor feels as if she is wielding the dynamics of the
surroundings so that she and the car are receiving the right road, and the lane stays centered around her and the car, and the other cars
flow off to her sides instead of into her.

29. Golanka (2015)
Controlling
action unfolding
continuously
over time
ACTOR AS SCULPTORACTOR AS CONDUCTOR
The industrial design also supports actor-as-conductor of information. A glance at a map to select the right road. A glance at the speedometer to
decide to switch her actions to slow down: she engages actor-as-sculptor behavior, relying on the shifting position of her foot on the brake pedal, and
the way the visual information is streaming through the windshield to control how she slows down continuously over time.
(NOTE: Experienced drivers are really bad at mimicking the act of driving. That’s because drivers don’t decide in their heads how to handle the steering wheel or adjust
the gas and break pedal positions. They rely on the continuous flow of information to help them choreograph the behaviors of slowing down and choosing the right
road and staying in the lane and moving around a car. Without this continuous flow of information to rely on, the meaning of the behavior is broken.)

30. ACTOR AS SCULPTORACTOR AS CONDUCTOR
Historically,
our field
lived here
Historically, our field lived here– actor as conductor– and actor as sculptor was the stuff of industrial design. For actor-as-conductor designs, there is of
course some physical action that has to happen to get the cursor on the pixels, or get the fingertip on the target, or say the words to trigger the
response of the system, but there were lots of degrees of freedom for how the actor controlled physical actions over time. Whether the actor was
sitting down, or walking, or reclining, and the arc and speed of the motions to work a mouse or tap a screen didn’t participate directly in the meaning of
the activity.

31. ACTOR AS SCULPTORACTOR AS CONDUCTOR
Historically,
our field
lived here
The thread of
meaning is
wandering here…
But, now the thread of meaning is wandering over here, to actor as sculptor.

32. ACTOR AS SCULPTORACTOR AS CONDUCTOR
Autodesk Alias sample car concept
Same with physical product design at smaller scales. Designers of physical things like cars and products are always looking for ways to
experience their designs before they are brought into the world in physical materials.

33. Feel human proportionality,
how designed objects will occupy surroundings
ACTOR AS SCULPTORACTOR AS CONDUCTOR
Autodesk Alias sample car concept; woman with Tilt Brush by Google;
Tilt Brush car doodle by wordpresshelp1234
They are experimenting with sketching in 3D with simulated light trails. They walk around in 3D drawing the geometry exactly as it will take up
space. They feel the curves, they can walk around the designs, feeling their proportionality to humans, how the designed objects will physically
occupy their surroundings. This is actor-as-sculptor of information: the way they control action continuously over time to draw the light trail is the
meaning. There are no degrees of freedom in their physical actions because the physical actions participate in the meaning directly.

34. ACTOR AS SCULPTORACTOR AS CONDUCTOR
Designs in Autodesk Revit and VR by Samuel Arsenault-Brassard
Building architects are experimenting with the handoffs across conducting and sculpting with information. This architect has detailed a building
in architectural software (conducting with information). He brings the building into 3D space in order to sketch some new concepts. He scales
the building down so he can walk around it and look down on it. He relies on information about the way the building occupies space to guide his
hand motions as he sketches new towers into the existing geometry. He can feel how the curves will occupy space with respect to the rest of
the building.

35. ACTOR AS SCULPTORACTOR AS CONDUCTOR
Designs in Autodesk Revit and VR by Samuel Arsenault-Brassard
Then, he can bring his rough sketch back into the software and do all the conducting (selecting, switching, styling) to detail the sketch and
rationalize it with the building.

36. ACTOR AS SCULPTORACTOR AS CONDUCTOR
Work with holograms as physical objects
MIT is experimenting with focused sonar that creates mechanical force against an actor’s hand. Now actors can do things like touch holograms,
and explore their surfaces and textures. An actor could work with holograms as physical objects, relying on haptic information to help control
physical actions continuously over time. (Image: MIT)

37. ACTOR AS SCULPTORACTOR AS CONDUCTOR
High-precision motion tracking
collapses degrees of freedom of
spacetime action into meaning
Image: Google Soli project
Google is using radar to very precisely track the position and motion of human fingers. The actor can mimic things like turning a dial, moving a
slider, pushing a button. These are still things for conducting (selecting, switching, styling), but there is a tiny bit of actor-as-sculptor: the actor’s
fingers become tactile information for each other to rely on to control action continuously over time. It’s just fingers now, but soon it will be hand,
arm, the actor’s entire body position at high precision that could participate in sculpting with information.

38. Actor as
Conductor
Selecting action, switching
action, styling action
Dynamics
of
surroundings
Concepts
INFORMATION
Physical
alignment
Semantic
alignment
MEANING
Golanka (2015)
Golanka (2015)
Actor as
Sculptor
Controlling action
unfolding over time
Structure of natural human behavior
Those are just a few examples of how we’re stating to need to worry about both human behaviors. Now I want to build out the structure of
natural human behavior for both of these. For actor as conductor, the meaning is semantic alignment: why we’re selecting an action, why we’re
switching action, why we’re styling the action. For Actor as sculptor, the meaning is physical alignment. For actor as conductor, the information
is about concepts; for actor as sculptor, the information is about the dynamics of the surroundings. Let’s look at what that means.
(NOTE: if you read this Golanka 2015 paper, I recommend as background, first reading her paper with Andrew Wilson, “Embodied cognition is
not what you think it is,” available in full from Frontiers in Psychology.)

39. This is not information for surfing
Underlying dynamics
This is not information for surfing. This is how wind transfers friction to water molecules, pushing and pulling them to form a wave. This is the
underlying dynamics that create a wave, but this is not information for surfing.

40. This is information for surfing
Invariant structure–the shape of the opening–creates affordances
for a surfer to stay inside this place made of liquid information
This is information for surfing. This expert surfer doesn’t see these dynamics, he sees the enclosure of a wave as a place to inhabit. And he aligns
his actions with information about the wave continuously over time to keep the enclosure around him. The shape and slope of the opening of the
wave is information that he can use to position the tip of his surfboard. He is calibrated to the way he must align his actions to this information; the
information becomes affordances for surfing for him. And when the shape of the opening changes, when he feels water hitting his back, he knows
that his place is collapsing and he needs to switch action, engaging other affordances, to quickly get out.

41. Skeletal muscle animation: Eleanor Lutz;
This is not information for goal keeping (soccer)
Underlying dynamics
This is not information for goal keeping for soccer. This is the underlying dynamics for how our muscles contract to move our skeletons.

42. This is information for goal keeping (soccer)
Point light image: https://quote.ucsd.edu/cogs91/
This is information for goal keeping for soccer. Twelve points of light are enough to give information to a goal keep about a human kicking a soccer
ball. The goal keep can align her actions to this information to prepare to stop the ball. She doesn’t have to have seen this particular human kicking
a ball, she doesn’t have to have seen ball kicking from this particular perspective in this particular light. She sees the relationships among how one
foot plants while another moves with respect to the knee. This becomes invariant structure, information about ball kicking, that work as affordances
to align with for a goal keeper. (NOTE: visit the URL above to see the animated gif to get the idea of this information.)

43. Information about concepts is different
Macro photo of Kindle Reader e-ink; vocal tract anatomy Bruno Dubuc
Point to meaning
by convention
FREEDOM
SPOON
(Not about underlying dynamics)
When we hear or read words like Spoon and Freedom, those things are not about the underlying dynamics of the chemicals or electrical
process creating the text, or the function of the larynx creating the sounds. These words point to meaning by convention. We have decided what
these mean in agreement spanning time and culture.

44. For actor as conductor, we assemble semantic alignment (why are we selecting, switching, styling an action?) with information about concepts by
way of convention. For actor as sculptor, we assemble physical alignment with information about the dynamics of our surroundings by way of
affordances. We see that conducting with information is not dependent on time. We can conduct in quick succession, or over weeks or lifetimes.
Time does not factor into whether conducting with information feels natural or not. For actor-as-sculptor, natural behavior is directly dependent on
being continuous over time. And that places some very important demands on the kind of information that can support actor as sculptor.

45. For actor as sculptor, information must be dense, persistent, and lawful. And it must be made of perceptual information. Language is not dense
enough, persistent enough, (though it acts as if it’s lawful across an actor’s lifespan) to rely on to control action continuously over time. The two
key things actors use perceptual information for to sculpt with information are: orienting & wayfinding, and prospecting with objects. We’ll look at
both of these.

46. Orienting & Wayfinding
Dense, persistent, lawful
Visual Information
Surfaces
Edges
Textures
Proprioception, Kinesthesia
We are always orienting ourselves physically in our surroundings. We humans are wired for proprioception and kinesthesia: we know where
our bodies are in space, and we know when we’re moving. We also rely on visual information for information about the relationships among
surfaces, edges, and textures in our surroundings that we can use as affordances for us to move: this path is walkable to me, these stairs are
climbable to me, this gravelly hill is not passable to me. And this information is dense, persistent, and lawful. We can rely on it to control our
actions continuously over time.

47. Wayfinding
Haptic Information
Dense (enough)
persistent (enough)
lawful
Simulates surfaces, edges
This woman has low vision and can’t rely on visual information for wayfinding. She’s wearing a device that simulates information about
obstacles as haptic information. The haptic information is much less dense than visual information, and is not persistent (it’s not a constant
flow, but kicks in with proximity to obstacles). But it’s dense enough, persistent enough, and lawfully specifies obstacles in her surroundings
that she can use as information for orienting and wayfinding.

48. Wayfinding
Sonic Information
Dense (enough)
persistent (enough)
lawful
Incoming cars
Train/no train
Crossing light timers
Conversations
Physical obstacles
Incoherent with “everyday sounds” for wayfinding
Gaver (1994)
UNNATURAL BEHAVIOR
This system can use sound instead of haptic information about obstacles. Sound is dense enough, persistent enough, and lawful, but the problem with
sounds is that it’s directional, but not spatial. It piles up. That’s great for music, but not for wayfinding. Haptic information can use spatial relationships to
represent multiple obstacles and their relative positions. Sounds just pile up. The other problem with sound fir wayfinding is that humans rely on many
other very important sounds: incoming cars, train/no train, crossing light timers, conversations. And the sounds about physical obstacles steps on those so
they are not dense, persistent, and lawful. This is unnatural behavior.
(NOTE: see Gaver 1994 in references for an excellent discussion of ecological taxonomy of “everyday sounds” humans rely on for information.)

49. So we see that we rely on many kinds of perceptual information for orienting and wayfinding, and we see that, in our designs to support this, we must
ensure the information we use is not only dense enough, persistent enough, and lawful, but coherent with the other perceptual information actors are
relying on for the activity. So, we’ll add coherence as a design consideration.

50. VR has a coherence problem
"I started forgetting which walls
were virtual and which ones were
real, and I ended up walking into a
few very solid walls"
“A player attempted to push
his hand through the floor to
sculpt something’”
Disrupts critical information for
placetime orienting and wayfinding
Dense (enough)
persistent (enough)
lawful
UNNATURAL BEHAVIOR
Virtual Reality also has a coherence problem. VR disrupts critical information for placetime orienting and wayfinding. Players of VR games report things like, “A
player attempted to push his hand through the floor to sculpt something.” Another: “I started forgetting which walls were virtual and which ones were real, and I
ended up walking into a few very solid walls.” These designs are mitigating this by adding in wall alerts and visual grids to show were physical walls are. These
things are dense enough, persistent enough, and lawful to represent the walls. The problem is that they are like aroras from another reality slicing through the
information from the present reality. They disrupt the dense, persistent, lawful information in the present reality for orienting and wayfinding and ask the actor to
align with two sets of conflicting information. That’s unnatural behavior. Quotes from: http://www.businessinsider.com/virtual-reality-gamers-are-literally-colliding-with-the-physical-world-2016-4

51. Image credit: Designboom (Alison Brooks Architects ‘The Smile’)
“One sees the environment not just with
the eyes but with the eyes in the head on
the shoulders of a body that gets about”
J. J. Gibson, founder Ecological Psychology
Gibson (1979)
The other problem with virtual reality is that we humans naturally need to explore our environment. JJ Gibson, the founder of Ecological Psychology
says…

52. Image credit: Designboom (Alison Brooks Architects ‘The Smile’)
Principle of action for VR:
“A participant must have affordances for moving about in
the scene…take action in the world and perceive the
effects. This is part of the larger sense of personal agency.”
Brenda Laurel
Laurel (2016)
Brenda Laurel has a principle of action for VR...

53. So, we’ll add explorability as a design concern.

54. Prospecting with objects
Image: https://quote.ucsd.edu/cogs91/
Color relationships become affordances for banana edibility
Surfaces
green : yellow : brown
We don’t just start using objects, we prospect with them using information about them. If we have experience eating bananas, the relative
amounts of green:yellow:brown color on the surface of the peel becomes information to us. We become calibrated to the relative colors to
know that a banana like this will have a peel that droops, we anticipate seeing brown spots on the fruit, we’d expect the fruit to be a little mushy
and sugary. This information becomes affordances that we can rely on about whether this banana is edible to us.
object affordances: Gibson (1979)

55. Prospecting with objects
Surfaces Edges
Textures
“smooth”
“glassy”
Relationships of surfaces, edges, textures become affordances
for finding my phone by dynamic touch
Curve of edges to face
End edges : side edges
Edge thickness
We can also prospect with objects using dynamic touch. I know my phone visually, but I also know my phone by touch. If I’m reaching into my bag to
retrieve my phone, I don’t just touch it once and know that it’s my phone, I must explore the relationships in the haptic information about the surfaces,
edges, textures of the phone. I understand that the surface is glassy and smooth. I understand the relationships among the end edges to the sides;
the edge thickness; the curve of the edges to the face. This haptic information becomes affordances to me to retrieve my phone.
dynamic touch: Gibson (1979)

56. Are our simulated objects explorable?
Do they know what kinds of objects they are?
Prospecting simulated objects
What kinds of information
should they simulate to reveal
new underlying dynamics?
New kinds of transformations
Permutational Computational
Smart Materials Same concern as simulated objects
Tactile/HapticVisual Sonic Same information as physical objects
What about prospecting simulated objects? We can simulate the same kinds of infomraitno that physical objects have: visual, tactile, sonic. But we can also
infuse simulated objects with new kinds of underlying dynamics that physical objects don’t have: new kinds of transformations. But, we also need to infuse
them with information about these new kinds of dynamics that humans can rely on. We need to start thinking about what a permutational affordance is like;
what a computational affordance is like. Smart materials have the same concern as simulated objects: smart materials are physical materials that are
infused with new underlying dynamics. How do we coax smart materials to radiate information about their underlying dynamics for humans to calibrate with
and use as affordances? (NOTE: We also need to ask, do we truly need a spacetime simulation? If the information is more abstract than the way the object orients and embeds in spacetime
and relative to the actors proportions, are we just making a 3D hologram versio of a skeuomorphism?)

57. Simulations have no mass!
Prospecting simulated objects
“There's no physical furniture
to break your fall when you
try to sit in a virtual chair.
Someone asked, Where did
that bruise come from?
I achieved presence.”
UNNATURAL BEHAVIOR “Breaking the ecological law”
Wilson (2012)
Yet…
they provide affordances for physicality COHERENCE
Tactile/HapticVisual Sonic
touch-ability, throw-ability, transformability
sit-on-ability, climb-on-ability
Yet…
one physicality affordance is not lawful
Visual
Explorability leads to a coherence problem for simulated objects. Simulations have no mass! Yet, they provide affordances for physicality, we perceive them as objects that
take up spacetime embedded in our surroundings. Get dense, persistent, lawful visual and haptic and tactile and sonic information, depending on how they’re wired up, about
touch-ability (surfaces and textures), pickup-ability, throw-ability, transformability, and so on as far as our imaginations take us. Yet, one aspect of their affordances for
physicality is not lawful. We get visual information for things like sit-on-ability, climb-on-ability. Surfaces and textures suggest the lawful presence of mass. These are
affordances we’ve been attuned to our whole lives, and we will engage with. One person in VR recently said: “There’s no furniture to break your fall when you sit on a virtual
chair. Someone asked, ‘Where did that bruise come from?’ I achieved presence.” Andrew Wilson, a present day Ecological Psychologist, would tell us we are breaking the
ecological law by simulating affordances for properties that our objects do not lawfully provide. That’s unnatural behavior.

58. True mixed reality: 3D scanned objects + simulated objects coexisting
Prospecting simulated objects
Scanned objects have
physical affordances
simulated objects don’t
How can we help
distinguish them so they
can coherently co-exist?
When we get away from this hard divide between virtual reality and holograms in “augmented reality,” and get to true mixed reality, we will have 3D scanned
physical objects co-existing with simulated objects. Scanned objects have physical affordances that simulated objects don’t. How can we help distinguish
them so they can coherently co-exist?
(NOTE: see Brenda Laurel’s reference for a description of mixed reality as a spectrum.)

59. So, that’s the structure of actor as sculptor behavior. Now let’s build out actor as conductor. Information to support conducting can take the form of both
language and perceptual information. Language for conducting is about coordinating and understanding. We’ll start with coordinating.

60. Ecological psychology tells us that natural human conversation is about coordinating. When someone speaks and someone listens, that evokes concepts.
Concepts point to meaning. Meaning points to values. And that’s enough to evoke a virtual array of cultural possibilities among the participants. Something as
seemingly here-and-now as asking which movie should we see tonight locates speakers and listeners in place and time, and evokes all of their placetime histories
as an array of cultural possibilities. Speaking is anticipating cultural possibilities by improvising wordings. Listening is aligning with the discovered nature of the
cultural possibilities, and entraining with rhythms, timing, pronunciation, grammar of the participants in the conversation. A conversation is a creative tension:
anticipating and aligning; improvising and entraining. The acts of conversation feed back into the structure of the cultural possibilities, changing the structure over
time. The overall behavior of this system of conversation is about coordinating actions among the participants to realize values.

61. Natural Human Conversation
Hodges (2014)
A creative tension in which participants
resonate to the larger story in which they
are participating, orienting and directing
them toward a good continuation
Coordinating action to realize values

62. Giant, ongoing, distributed conversation with highly refined
entrainment of grammar across its placetime histories
Many overlapping conversations with participants resonating
to very different structures for good continuation
When someone that aligns with a particular cultural structure for good continuation encounters acts of conversation that resonate to a very different structure for
good continuation, there is nothing to align with. The array of cultural possibilities is invisible to both parties. Is there any way for us to visualize these structures to
help facilitate some kind of understanding (if not aligned values) across conversations?

63. Conversation
If you want to visit virtual reality, just speak
to someone, and listen, with intent toward
a good continuation
A virtual semantic array made of your
collective histories of culture at every
placetimelevel will emerge all around you

64. So, we see that natural human conversation is about coordinating. Now let’s look at using language for understanding.
(NOTE: I am using “understanding” as the label here to mean the notion of design for understanding in Morville, Rosenfeld, Arango (2015).)

65. When an actor engages a design that supports something like shopping, banking, creating, hailing/styling a service, etc., the actor is engaging with language based on classification: the underlying
information architecture of the designed system. And the purpose of the behavior is understanding/performing for a good continuation. The actor engages the design and that evokes concepts.
Concepts point to meaning. Meaning points to values. And that’s enough to evoke a virtual array of classificational possibilities. This takes the form of the underlying system of information architecture.
The actor anticipates the performance of the classification by improvising selecting/switching/styling. Selecting/switching/styling can take the form of browsing, searching, commanding, hailing,
parameterizing, etc., whatever the design facilitates. The actor then aligns with the meaning of the performance of the system (discovered nature of the classificational possibilities). And entrains with
the labeling and grammar of the system. Entraining is vital here: if the actor does not entrain to the labeling and grammar of the system, the virtual array of meaning falls apart. We know from our field
of IA that a system must be clear, and resilient across channels and contexts. The behavior of the system is about understanding/performing for a good continuation.

66. Now we are playing with the parts of this system: autonomous digital agents offload from the actor the tension of anticipating/aligning with the classificational possibilities. The
agent does this work. But, when the agent needs to interrupt the human with a decision point or status, the agent must bring the right kind of information to let the human
anticipate/align with the digital agent for a good continuation. We are also injecting machine learning into the system. That can greatly expand what is allowable for entraining
with the system. Entraining is still vital, but the human has more flexibility in what is allowed for labeling and grammar. Machine learning in which the system is expanding the
nature of the underlying classificational possibilities will expand the scope of what the human needs to anticipate and align with. I think that things like machine learning and
conversational UI are really about expanding the tolerance for imprecision the human must have to engage the system.

67. Human-System imprecision tolerances
NARROW Tolerance
Precise entrainment
WIDE Tolerance
Forgiving entrainment
Natural Language Processing
Machine Learning
(entraining actor’s wordings)
Conversational UI
Standard searching, voice commanding
Gestural UI
(degrees of freedom of motion, subtle vs elaborate)
For any given system, the system has a tolerance for imprecision in what is needed to engage it. A narrow tolerance for imprecision would be a system that requires very
precise entrainment with labeling and grammar. A wide tolerance for imprecision would be a system that has forgiving entrainment with labeling and grammar. Something
like standard searching and voice commanding have narrow tolerance for imprecision. We can do things like autosuggest, and spelling forgiveness, but these are generally
very precise labeling and grammar requirements. Conversational UI expands the tolerance for imprecision by suggesting the classificational possibilities ahead; this gives the
actor a hint of the performance of the system as she is selecing/switching/styling. Natural language processing expands the tolerance for imprecision. Machine learning in
which the system entrains on the actor’s particular wordings greatly expands the tolerance for imprecision with a very forgining entraining requirement on the human.
Something like gestural UI right now has a narrow tolerance for imprecision. I believe the tolerance for imprecision should be much more forgiving to mime concepts, both in
terms of the degrees of freedom of motion to give the system the gist of the concept, and in whether the human can use subtle motions or must do elaborate gestures.

68. So, we’ll add imprecision tolerance as a design consideration. For actor as conductor, we can also use perceptual information. This takes the
form of mapping: infusing concepts with perceptual qualities in the form of visualizations and notifications: maybe a sound or haptic buzz or color
change to signal a state change.

69. Mapping
Air flow and heat transfer simulations, Autodesk
Visualization Values-realizing, good continuation, clear, resilient
Curvature mapped to airflowColor mapped to temperature
This is a visualization of the air flow and heat transfer in a hospital operating room on the left, and an airport walkway on the right. Color is
mapped to different temperature ranges. The arc of the lines shows the direction of the airflow. We could describe something like this in words,
but this visualization captures the ambient comfort of these places as a quick, visceral understanding. That’s because perceptual information
and language feel different in terms of the ease of the flow of engaging them.

70. Perception flows easily
Tacit
Reflexive
photo credit: Jason Pratt via Creative Commons
Language is viscous,
more work to flow
Awareness
Associativity
Requires attention
These kinds of information have different viscosities, or ease of flow.

71. Perceptual information
Language
Mercedes-Benz F 015 Concept Car
VISCOSITY: ease of flow of conducting
visceral
Low viscosity, visual information about
density, directionality, color
Situational visualization human can manipulate?
intense
concentration
conceptual
High viscosity, full entrainment
with author’s wordings
Viscosity with precise grammar/label
entrainment, linear over simultaneous
Self-driving car handoff to human…?
How will the car interrupt us?
How will the car map sculpting
information to concepts?
intense
coordination
Something like the airflow visualization is all perceptual information, no language. It is a very visceral way to understand with information. Something like reading a novel is all language, high
viscosity entraining with concepts and the author’s particular wordings. It is a conceptual way to engage meaning and requires active attention. Something like talking to Alexa also is all language,
and requires precise entrainment with the grammar and wordings. It is also high viscosity because it requires linear order (you can only do one thing at a time). What about when we think about
the interaction among a human and a self-driving car? The car is offloading all of the sculpting behavior of driving a car. What about when the car needs to interrupt the human for a critical
decision? How will the car sum up all of the perceptual information it has been aligning with to control the actions of the car continuously over time into concepts to align with humans? Will it use
wordings? Will it offer an ongoing visualization of the situation that the human can manipulate for a decision point? Like a flight tracker the human can tweak? We need to worry about the form
that information will take in this kind of important handoff between sculpting (the car) and conducting (the human). NOTE: I fear my brief examples seem to imply language viscosity is negative.
On the contrary! Please see my 2015 IAS talk for a more detailed treatment of viscosity: https://www.slideshare.net/mjane_h/what-we-mean-by-meaning-new-structural-properties-of-information-architecture-
ias15-47485499?qid=95845315-8b4c-4c4b-8949-48b464ba8547&v=&b=&from_search=5

72. So, we’ll add viscosity as a design concern. We see that the structure of natural human behavior for actor as sculptor is physical alignment using the
dynamics of the surroundings by way of affordances. It relies on perceptual information for orienting & wayfinding, and prospecting with objects. Perceptual
information for sculpting can take many forms, but must always be coherent with the way other kinds of information are being used for the activity.
Information for sculpting must be dense enough, persistent enough, and always lawful. Natural human behavior for actor as conductor is semantic
alignment using concepts by way of convention. Both language and perceptual information can be used to support conducting. Language for coordinating
takes the form of human conversation. Language for understanding takes the form of systems of classification. Perceptual information for conducting is a
mapping to concepts using visualization or notification. Information to support conducting must be values-realizing, a good continuation, clear, and resilient.

73. ACTOR AS
CONDUCTOR
Controlling action
ACTOR AS
CONDUCTOR
ACTOR AS
SCULPTOR
Which behavior is our design supporting?
Degrees of freedom in controlling action Action unfolding continuously over time
participates in the meaning

74. Earth image: NASA Apollo 17 crew https://upload.wikimedia.org/wikipedia/commons/thumb/7/78/The_Blue_Marble.jpg /1023px-The_Blue_Marble.jpg
Follow the thread of meaning
wherever it may wander
As designers and information architects, we follow the thread of meaning wherever it may wander.

75. Natural doesn’t have to be familiar
Image: Hugh Hochman
Natural doesn’t have to be familiar, or even physically present. Nascent technologies will always race around, and we should
knock ourselves out experimenting with the possibilities.

76. Image: DMITRIS_K Big Think
Disrupting the ecology of information
If our designs feel deep-down unnatural, and we can’t pin it on usability or usefulness or lack of clarity, or any of our other
ways to say what’s good, maybe it’s because we’re disrupting the ecology of the ways our human actors are relying on
information to behave.

77. Human behavior is events aligning,
Tuned to align our actions
For 100s of 1000s of years, and more across the branches of the tree of life, our perceptual systems have been tuned to
align our actions…

78. with information around us
…with information all around us.

79. with information around us
(Not the underlaying dynamics causing the information.)

80. with information around us
But the information. We align our actions with information around us...

81. to realize our values
…to realize our values.

82. All human behavior is information behavior
From an ecological psychology POV, we see that all human behavior is information behavior.

83. Marsha Haverty
Thank you
@mjane_h
2017 Information Architecture Summit

84. References
Bates, Marcia (1989). The Design of Browsing and Berrypicking
Techniques for the Online Search Interface
Sagan, Carl (1994). The Pale Blue Dot
Golanka, Sabrina (2015). Laws and conventions in language-
related behaviors, Ecological Psychology 27:236-250.
Gibson, James J. (1979). An Ecological Approach to Visual Perception
Marsha Haverty @mjane_h
Hodges, Bert (2014). Righting language: the view from
ecological psychology. Language Sciences, Jan
Cowley, Stephen and Harvey, Matthew (2015). The illusion of
common ground, New Ideas in Psychology p. 1-8.
Wilson, Andrew (2012). Breaking the (ecological) law: why
illusory sounds don’t make for safer cars. Online
http://psychsciencenotes.blogspot.com/2012/10/breaking-
ecological-law-why-illusory.html
Barrett, Louise (2011). Beyond the Brain: How Body and Environment
Shape Animal and Human Minds, Princeton University Press
Arora, Rahul et. al (2017). Experimental evaluation of
sketching on surfaces in VR. CHI 2017, May 6-11 2017.
Founded Ecological Psychology
Science in
contemporary
Ecological
Psychology
Laurel, Brenda (2016). What is Virtual Reality? Published
online on Medium, 15 June
Thibald Paul J. (2011). First-Order Languaging Dynamics and
Second-Order Language: The Distributed Language View.
Ecological Psychology 23(3): 210-245.
Gaver, William W. (1994). What in the world do we hear? An ecological
approach to auditory perception. Ecological Psychology 5(1): 1-29.
Covert, Abby (2014). How to make sense of any mess.
CreateSpace.
Hinton, Andrew (2014). Understanding Context: Environment,
Language, and Information Architecture. O’Reilly Media.
Resmini, Andrea and Rosatti, Luca (2011) Pervasive
Information Architecture. Morgan Kaufmann.
Rosenfeld, Louis, Morville, Peter, Arango, Jorge (2015).
Information Architecture for the Web and Beyond. O’Reilly Media.