I had the pleasure of speaking at Interact London in October 2014. I presented an updated version of this talk, which I originally gave at IA Summit earlier in the spring. The talk is based on content from my book, Understanding Context. You can read more about it at http://contextbook.com. In this version, I have updated the way I'm talking about how language works as environment: instead of 'semantic affordance' I'm now calling it 'semantic function.' (Which is in keeping with how it's now being described in the book.)

1. LANGUAGE IS
INFRASTRUCTURE
Thoughts on the material of IA practice (updated)
PRESENTED BY
Andrew Hinton | @inkblurt | InteractConf London 2014

2. HI. I’M ANDREW.
AND I’M OBSESSED WITH CONTEXT.
2
contextbook.com
(Due December 2014)
#shamelessplug
Hi I’m Andrew and I’m obsessed with context.
In fact, I’m finishing a new book on the topic, due out this December.
Writing this book was an amazing learning experience for me, and one of the things I learned the most about was the
role language plays in the work we do, and in the environments we design for people.

3. 3
“Make it easier to use the shopping cart.”
Here’s a hypothetical example most of us can relate to.
Let’s say you’re doing user-experience work for a retailer’s e-commerce site.
The project you’ve been assigned has what sounds like a clearly defined goal: Make the shopping cart easier to use.
Ok, great. We all know what the cart is -- it’s been around on websites for a long time now.
But is it really so clear-cut?

4. OK, BUT ... WHAT IS “CART”?
4
For starters -- what is the “cart”?
It’s the kind of question that would seem to invite eye-rolls and derision ... what do you mean what is “cart”? It’s
right there on the website!
http://upload.wikimedia.org/wikipedia/commons/archive/5/50/20110713132415%21ShoppingCart.svg

5. WILL THE REAL CART PLEASE STAND UP?
5
Cart isn’t just one thing ... sure, there’s an icon in the top right of the website. Ok.
>>
But the cart actually shows up in many places on the site, in various forms, and enmeshed with all sorts of other
functions, like authentication, store choice, product availability, and shipping information. And that’s before we even
get to checkout.

6. 6
What parts of “cart” do you mean?
=
Business rules
Software services
Interfaces
Serial processes
User behaviors
All of these involve language.
The cart isn’t just the cart -- it’s a signifier we use as shorthand for a hugely complex cluster of business rules,
software infrastructure, not to mention interactions and user behaviors.
>> And all of these involve, or are made of, language.
The only way to work this out is to have a conversation about it, unpack the parts, and gain some consensus or
direction on what the specific mission is. This is a simplified example, but I know I’m not the only one in the room
who has encountered similar scenarios.
I’ve worked with retailers who’ve actually organized their teams based on parts of their website: so what would the
“Cart and Checkout” team actually work on?
It would be great to do the work of digging into the meaning of what we’re working on... but more often than not
there’s not time or willingness to stop.

7. 7
Physical Object
Virtual Object
Design is
obsessed
with
tangible
objects...
Design is obsessed with the tangible.
Everything else feels secondary, peripheral. Make the object, and we’ll name it later. Stop talking about it and start
making stuff.
But what’s missing here is the context that makes these physical and virtual objects valuable or meaningful, the stuff
that connects them and defines them.
http://commons.wikimedia.org/wiki/File:Samsung_galaxy_Note_-_Snap_2567.JPG

8. 8
“That’s just semantics...”
A phrase I hear a lot is “that’s just semantics.”
>> And that, right there, crystallizes the problem. That word “just”... think of all it implies: That semantics is small,
inessential, a deviation of purpose.

9. 9
Like this fish and its relationship with water, we are so immersed in language -- it is such a native medium for
human beings -- that we see right through it, and take it for granted.
But it is a medium, and it does have properties that we should understand.

10. Language is a real medium for making.
10
Today, my talk really has just one message I hope you take away: language is a real medium for making.
That semantics isn’t fluff, it isn’t inessential. Instead, it’s crucially essential, and central.
It's not a disembodied layer of abstraction that we optionally add to the objects, places, and systems we design. It is
instead both the body and soul of those objects, systems, and places.
In particular, language is what we use when designing the structure of information environments. That is, when we do
information architecture. Language makes structures that are as real and critical to our lives as the walls, ceilings, and
floors we see around us in this room.
http://commons.wikimedia.org/wiki/File:Kneden_van_klei.jpg

11. 11
ORGANIZATIONS
Let’s start by looking at how semantics function as part of the environment of organizations.

12. LANGUAGE MAKES MAPS THAT ARE TERRITORIES
ENVIRONMENTS WE INHABIT TOGETHER
12
Organizational diagram of the New York and Erie Railroad, 1855
Organizations depend on language as a sort of map that they live in as a territory.
This gorgeous, early org chart from the 1800s is a fascinating example of early attempts to show structure before the
mechanical, engineering metaphor had fully infiltrated the way we think of organizations. But either way, org charts
are representations of social relationships, processes, collaborative structures. This railroad company, spread out
geographically over space and time zones, needed an architecture that virtually organized its relationships -- an
augmented-reality architecture for existing as a corporate entity.

13. 13
REQUIREMENTS & STORIES ARE MOLDS FOR MAKING
When we create requirements or user stories, we’re putting a lot of weight on language to define the shape of what
we will produce. Even though the language isn’t the product, it casts the product in its image.
card: http://itsadeliverything.com/agile-requirements-snail-feature-to-user-story-to-scenario
casting: http://tombanwell.blogspot.com/2010/09/olifant-neoprene-trunk-hose.html

14. HOW WE TALK ABOUT THE WORLD
FINDS ITS WAY INTO OUR SYSTEMS & ENVIRONMENTS
14
“These different terms have
slightly different connotations.
Whatever terminology we use in
our architectures, it needs to
reflect these connotations.”
Jesse James Garrett
“Brand Driven Information Architecture”
February 2004
Ten years ago, at the IA Summit, Jesse James Garrett talked about how an organization’s brand depends on the
foundational meanings of the language it uses, and that the structures we make need to take this language seriously.
How we talk about the world will find its way into our products and communications whether we’re paying attention
to it and bringing the necessary rigor to it or not -- so, best to do it on purpose.

15. 15
CULTURE IS A CRITICAL BUSINESS CONCERN
AND OUR INTERFACE WITH IT IS LANGUAGE
from The First 90 Days, Michael D. Watkins
In the classic new-management advice book, the First 90 days, the author stresses that to work effectively as a
manager or executive, you have to understand the cultural norms and fundamental values of the organization.
Management has as much to do with the language of the company as with the budgets and reporting employees.

16. ORGANIZATIONS HAVE STORIES
THAT DRIVE THE CONVERSATION ABOUT WHAT IS MADE
16
Underlying Story
Interface
Organizations have collective memories, and shared narratives. The arbitrary pathways of the past are often
remembered as a linear story, that can be very convincing, because it leaves out all the paths that weren’t taken. And
that story can influence future decisions.
>>Likewise, even on a project level, that underlying story ends up being articulated in terms of interfaces.
“We need to add another tab to our global navigation” or “we need an app” or “we need to be more like pinterest” are
often the best way stakeholders know how to explain their problem.
The interface is a semantic construction that plays a poor proxy to defining the underlying problem to begin with.

17. INFORMATION THAT REQUIRES ARCHITECTURE
VERTICALLY & HORIZONTALLY
17
Language plays a critical role both vertically and horizontally in organizational life.
Vertically, it starts as deep as the data architecture. Data structures are pure language, structured and defined for
machines. Get something defined wrong or conflated down in a database entity structure, and it can cripple an
organization’s effectiveness.
There are the software services that connect the data with everything above. Those, too, are developed based on
definitions and requirements or user stories that were discussed, composed, and organized into priorities and
structures.
Then there are business rules -- the “if this then that,” cause and effect definitions that do the work of making a
company’s business model be an active, working mechanism in the world.
Of course, there are also what I’m calling here “corporate semantics” -- a catch-all term for how the organization
culturally talks about what it does, the narratives in the organization, and even the acronyms and taboo phrases.
Then all of this bubbles up to customers and end-users, in various interfaces -- digital or analog.
Horizontally, everything in the vertical stack touches all the various channels that the organization needs to manage,
from websites to product names, to customer support scripts and store signage.
Information architecture has a role to play in all of these areas, because as a practice it’s about architecting coherent
structures across all of these contexts. Systems that make sense.

18. 18
ENVIRONMENT
So language has a real, tangible effect on the work we do, and the organizations for whom we do it.
How can that be?
It’s because language functions as a real part of our environment.

19. 19
Language is
physical.
Language is physical. We make it and consume it with our bodies. When we speak, we’re using a great deal of our body to breathe and
create the sound vibrations needed for articulation, not to mention gesturing and “body language.”
When we write, we’re adding physical information to the environment, that we depend on being picked up and interpreted correctly by a
reader.
We learn to read by reading aloud, so even when you read to yourself silently, your body still subvocalizes, firing neurons as if your body
were making the sounds. In fact, NASA has used this subvocalized activity in new technologies that allow users to give commands without
having to literally say them⁠4.
Silent reading: http://neuro.hut.fi/~jjkujala/pdfs/perrone_bertolotti_et_al_JNeurosci2012.pdf
NASA story: http://www.nasa.gov/home/hqnews/2004/mar/HQ_04093_subvocal_speech.html
http://commons.wikimedia.org/wiki/File:Larynx_(PSF).png
http://commons.wikimedia.org/wiki/File:Witten_Blackboard.jpg

20. 20
There’s mounting evidence that language has been with our species long enough to be a factor in natural selection — part of the
environment that shaped our evolution. For example, anthropologists have shown there’s a connection between language use and the ability to
create sophisticated tools and weapons. This stone arrow tip dates to many thousands of years before the emergence of homo sapiens.
Image/article: http://www.theatlantic.com/technology/archive/2013/12/researchers-discover-the-hot-new-technology-throwing-javelins/281976/
Spear: http://commons.wikimedia.org/wiki/File:Spear_(PSF).png

21. LANGUAGE IS ENVIRONMENT
21
Language is “a form of mind-transforming
cognitive scaffolding...”
... “Simple labeling adds a new realm
of perceptible objects.”
- Andy Clark, Supersizing the Mind
http://en.wikipedia.org/wiki/File:CantileverScaffold.jpg
Language is environment -- it’s stuff we put into the world that allows us to make more stuff, including whole
civilizations.
Philosopher and writer Andy Clark, in his book on embodied and extended cognition, talks about how language is a
kind of scaffolding. We create it so that we can go on to create more environment together.
Likewise, names and categories allow us to rearrange the world. And they also add new structures to the
environment, new objects and places.

22. 22
A name is like
a TARDIS.
Small on
the outside;
huge on the
inside.
Labels are like Tardises. Innocuously small and mundane on the outside, but immensely powerful, mysterious, and
huge on the inside. Labels carry with them much of what the world is to us.
They allow us to rearrange the world and move whole universes of meaning with a breath or a scribble.
http://commons.wikimedia.org/wiki/File:TARDIS-trans.png

23. 23
“The list is the origin of culture.”
- Umberto Eco
And when we put labels into structures, such as lists, we create new parts of our environment that enable new kinds
of human activity.
The list, as Umberto Eco has said, is the origin of culture.
In archeological digs, lists are the most common sorts of writing found. Most of them are about commerce and
mundane recordkeeping. Why? Because writing was a necessary infrastructure for markets to expand over space and
time, past the natural limits of human memory.
1 http://www.newyorker.com/reporting/2007/12/10/071210fa_fact_gawande
http://commons.wikimedia.org/wiki/File:List_of_possessions_of_a_temple_AO8110_mp3h9037.jpg
Eco quote: http://www.spiegel.de/international/zeitgeist/spiegel-interview-with-umberto-eco-we-like-lists-because-
we-don-t-want-to-die-a-659577.html

24. 24
“Taxonomies provide the lenses by
which we perceive and talk about
the world we live in.”
- Patrick Lambe, Organising Knowledge
As Patrick Lambe says in this really amazingly wonderful book, Organising KNowledge, “Taxonomies provide the
lenses by which we perceive and talk about the world we live in.”
Keep in mind, taxonomies aren’t just hierarchies; taxonomies can be lists, spectrums, facets, polyhierarchies, and all
sorts of arrangements of language that create environmental structures for action and understanding.

25. The perceived functional properties of objects, places
25
AFFORDANCEand events in relation to an individual perceiver.
Physical
Affordance
Semantic
Function
James J Gibson, who invented the concept of affordance, talked about how the structure of the surfaces and objects
in our environment has a direct, mutually coupled relationship with our bodies, which pick up information about what
our bodies can do with those structures.
So, these stairs are picked up by our perceiving bodies as a structure that will take us upward in space if we climb
them.
But we don’t know where it’s going to take us. We don’t know the contextual significance of the stairs unless we go
up them and find out for ourselves, or someone tells us.
>> So the intrinsic affordance of the stairs is supplemented by a label -- something I was until recently calling
“semantic affordance” but I’ve changed. I’m now calling it “semantic function.”
I like this phrase “semantic function” because it helps remind us that even though it’s not strictly an affordance, it
still isn’t just an ephemeral bit of vapor — language has physical properties, and physical significance. It’s as much
part of our bodies as anything else; it functions as part of the machinery of our environment.
[longer bit i’m skipping]
Why? Because I realized that calling language an affordance wasn’t really in keeping with how Gibson framed the
concept. Affordances aren’t mediated in the way language is, and they don’t require convention for meaning. Stairs
don’t need to be interpreted; they will work the same for a body whether that body speaks English or Chinese. Calling
language an affordance can mislead us into thinking language can be intrinsically meaningful, when it’s actually the
opposite. Language is made of symbolic information, so it’s intrinsically ambiguous. It requires even more context.
Donald Norman calls these signifiers -- and they are signifiers, no doubt -- but signifiers are ultimately grounded in
the same physical and cognitive mechanisms that we use to climb stairs, or walk through doors.

26. 26
Semantic
Function
Physical
Affordance
This distinction between semantic function and physical affordance is not meant to say they are binary opposites. I think they are intimately connected, part of
something more like a continuum than separate categories.
The truth is, people perceive all of it as one environment. But for purposes of designing these elements, it’s important for us to know the differences and have
names for them. This is part of understanding our materials.
JJ Gibson touched on language and cultural systems in his work. When developing the theory of affordances, explained that affordance isn’t only about simple
perception between my body and a surface; he argued that affordances between people -- because of language in particular -- create extremely high levels of
behavioral complexity and meaning.
These affordances coalesce to become compounded structures in our environment that are cultural constructs rather than physical ones.

27. 27
Semantic
Function
Physical
Affordance
Gibson talks about how a mailbox affords mailing a letter to another place. The intrinsic structure of the mailbox is just a hollowed out object that affords putting
small objects into. But for people encultured in a complex system of postal mail, we perceive it as something beyond its immediate physicality. In the terms I’m
framing here — The immediate physical affordance of the mailbox is an object to put something into. But semantic function allows us to know this is a special
box that is connected to other systemic physical affordances beyond our perception — mail trucks, postal offices, mail handlers.
>>What software does is create simulated versions of these affording systems — made entirely out of semantic information, which is what we use to make
programming languages and network protocols. It abstracts the compound affordances of “mail” into something digital machines do for us. So we have to use
visual and textual metaphors to represent what was physical in our world. An email inbox is not a box. It strips away the physical affordance and leaves us just
with the semantic function — which requires even more care and attention to definition and context.
http://commons.wikimedia.org/wiki/File:Post_office_drivethrough_lane.jpg

28. 28
PLACEMAKING
&
SENSEMAKING
A couple of important dynamics that we depend upon language for every day are placemaking and
sensemaking.

29. 29
When you see a city scene like this, you see placemaking happening everywhere, through the signs and labels on
buildings, but also through the conversations, publications, arguments, and cultural narratives.
Language certainly does work as a sort of scaffolding that enables us to build and live in places like this. But what I
actually see here is also the reverse relationship: a lot of infrastructure that is built to supplement the conversations
of culture and commerce. Perhaps the built environment is here to better enable the semantic environment, more
than the other way around. After all, we’ve been users of language longer than we’ve been builders of cities.
http://commons.wikimedia.org/wiki/File:Taipei_City_Nanyang_Street_20130127.jpg

30. WHAT ONCE WAS SPACE
IS NOW A PLACE
wikimedia.org
30
All it takes for humans to make something into a place is to say something about it. To name it, or mark it in some
way.
There were names for craters and mountains on the moon long before anyone walked on it.
But when the first humans actually set foot on the surface, they marked it with this flag.
No text is on it, but it’s still a rhetorical act. A semantic utterance.
The official line was “we come in peace, for all mankind.” But the subtext was, “Hey Russia, look what we can do.”

31. 31
“A one-star increase in
Yelp rating leads to a 5-9
percent increase in
revenue.”
- Michael Luca, Harvard Business School
Working Paper
Digital networks make semantic environments a super-charged, geometrically more immersive and influential
infrastructure that ends up changing the physical realities of behavior and place.
Yelp adds a meaningful layer to the environment that changes the sorts of places these are -- it has a real, physical
effect that has increased business to independent restaurants, taking it away from the chains, because now there’s
information in the environment that customers can use to feel comfortable trying a new place.
http://www.hbs.edu/faculty/Publication%20Files/12-016.pdf

32. 32
LABELS AND CONNECTIONS
CREATE PLACES
Poetry room photo http://www.flickr.com/photos/weanders/3675931616/
In bit-based digital environments, we have very little physical structure to help us clarify what our language means --
we depend on the language for all of our structural affordances. The stairs to the poetry room and the poetry room
itself conflate into the information inferred by the label.
This was the generative problem that spawned web-oriented information architecture. How to organize space where
everything is made of language, without the intrinsic structures of physical place to guide us.

33. WHAT PLACES ARE INSTANTIATED HERE?
THE LABELS, CONNECTIONS, AND RULES ARE MUDDLED
33
So when the labels don’t make sense in relation to one another -- when the labels, connections, and rules are
muddled, we’re disoriented. Sensemaking is stymied, and placemaking is dissonant.
In this example from the Facebook mobile app, you can choose what mode you prefer for your News Feed. Each
mode instantiates a different sort of place. But what are these places. There’s a News Feed inside of News Feed, and
a Most Recent option, which one might normally assume is also “News”. Then there’s All Friends, which is really
confusing -- are the other feeds not including all my friends? How are these structures nested in relation to one
another? There are hidden rules under these confusing labels that the environment doesn’t make structurally clear.

34. THE “CLOUD”
34
I’ve become less and less enamored with the language we use for “the cloud.”
A cloud is pretty from the outside, but from inside it’s just fog.
Cloud is a semantic abdication of responsibility. We understand our environment and how it works because of the
seams between surfaces, the structures we can perceive.
If the cloud were truly magical and just omnisciently knew how to behave and work based on our specific, unique
context moment to moment, then fine.
But the cloud actually has seams and surfaces, but we do a terrible job of exposing those structures to users so they
can understand the way those structures work. We end up posting things where we shouldn’t, erasing files on a team
dropbox account, or not realizing our private photos are being saved on Apple’s iCloud, where hackers can get to
them.

35. CHANNELS STITCHED TOGETHER WITH LANGUAGE
35
My Recipe Box
- Desktop +
Mobile
Aisle
Wayfinding +
Mobile
Labeling
Alignment
ID number - semantic marker
that ties everything together
The ways we enable and influence sensemaking and placemaking aren’t just about a single website or app either -- it
has to do with all the contexts that we encounter in something like a grocery store service experience. At the
Wegman’s grocery chain, there’s an architecture that uses language to stitch together the relationships between a
physical ID card and a digital one, and how that identifies you to the infrastructure that tracks your purchases, helps
you build a recipe plan and shopping list, and find a specific kind of peanut butter in your store.

36. INFORMATION ENVIRONMENTS
36
Consider the whole ecosystem of information environments that make up a retail customer experience. There’s the
conversations a customer has with her family, with friends and neighbors, the social media discussions and customer
reviews. There’s the published reviews by experts, price comparisons across the web, and the brand messaging
coming from media advertising.
Not to mention all the language one encounters in a retail store, from the labels on the aisles to the conversations
with service people.

37. 37
SYSTEMS
Language is both our interface with complex systems, and our material for making them.

38. THE STARING GAME
A SEMANTIC FRAMEWORK FOR SOCIAL BEHAVIOR
38
*blink*
HA!
I WIN!
“First one to blink loses”
Think about how the staring game works.
If I just walked up to anyone and started staring, I’d likely be thought rude.
But all it takes is asking “hey want to play the staring game?” or “I dare you to keep staring and not blink - whoever
blinks first loses” and you’ve created a sort of shared map of structure between you that changes the place you’re in,
if only for a few minutes. Architecture that’s invisible, made only of an agreement.
(example adapted, with thanks, from part of Frederik van Amstel’s presentation at the 2012 Interaction conference in Dublin)

39. BASEBALL WOULDN’T WORK...
... WITHOUT THE STRUCTURES OF LANGUAGE
39
http://en.wikipedia.org/wiki/File:Baseball_diamond.svg
All games work this way. They all have structures made of words.
This is a diagram about baseball, but the same principles apply to cricket… just watch a game and think about all the
action that happens on the field that is guided by a collective agreement to follow a set of rules.
There are actually very few physical structures constraining action on a baseball field, no rails connecting bases, no
apparatus taking a batter off the plate after three strikes. People just do it, because they agreed on the rules. Invisible
structures mapping a sort of temporary shared hallucination of “Place” for the sake of the game.

40. SWEDEN, 1967, THE MORNING AFTER THE RULE CHANGED
FROM DRIVING ON THE LEFT SIDE TO DRIVING ON THE RIGHT
40
When rules change without everyone being on board, bad things can happen.
This is Sweden in 1967 the day after a law passed that changed the legal side of the road for driving.
The same streets were there, but just a change by a legislative body created structures in the environment that not
everyone understood yet.
All these things are just made of language, and yet they create, shape, and change Places for us all the time.
This is basically what happens on Facebook every time they update their privacy controls.
picture: wikimedia commons

41. 41
So, business rules, systems, maps, environments -- all dependent upon and are inseparable from language.
That means, we have to take language seriously, and understand the nature of this material and the infrastructures it
influences and creates.
I foresee a near future where design schools and information management schools require students to study
linguistics and semiotics, to have an understanding of all the layers that make up how language works, the same way
we need to understand how CSS relates to HTML, or how a data stack relates to a presentation layer. Because,
ultimately, all of those things are made with language too.

42. 42
THREE THOUGHTS...

43. 43
MODELING IS MAKING
In a previous article I wrote on this topic, I argued that planning is making -- and I still think that *good* planning is
making. But good planning involves modeling: taking tacit abstractions and complex entities and dynamics and
making them into objects that we can work with in time and space.
Good modeling is a kind of prototyping -- but you’re prototyping your understanding of the problem and,
eventually, of the solution. It’s not valuable in and of itself; it’s valuable as a way to achieve a collective
understanding and a shared vocabulary. Scaffolding for collaborative knowing.
This is hard work. It is design work. And whether it’s a month of workshops or a five minute conversation over a bar
napkin, it is important work.

44. BUSINESS RULES
ARE EVERYONE’S
RESPONSIBILITY
44
All of these systems are built up from what companies call business rules. And I want to say clearly that these
business rules are everybody’s responsibility.
Never assume that just because something is a business rule that you don’t have something important or helpful to
say about it.
As our practice matures, we need to be able to participate in the discussions that determine the nature and overall
direction of our work. Business rules are a story the business is telling itself; it’s often up to us to dig underneath
those stories and figure out what’s driving them.

45. IT’S LANGUAGE
ALL THE WAY
DOWN
45
Third - there’s really nothing that matters to us as humans that isn’t somehow wrapped up in language.
The ancients who first used the word “poet” were using a word that meant “maker” -- because they understood that
the poet makes more than poems. The poet makes worlds.
We are a linguistic species -- grit and gristle halos and horns -- immersed in symbol, and suffused with story.
It’s language all the way down.

46. 46
THANK YOU
ANDREW HINTON | @INKBLURT
THE UNDERSTANDING GROUP