The document discusses how direct manipulation breaks down in the context of IoT devices. Using connected devices conceptually requires thinking like a programmer by requiring users to anticipate future needs, plan solutions, configure systems to meet needs, and deal with unexpected conditions - tasks that are difficult for non-expert consumers. While empowering users to program could be seen as positive, it requires a significant time investment that many users may not have. The document explores ways to make controlling complex connected systems easier through better feedback, visibility into remote contexts, explaining unexpected behaviors, and error prevention.

1. Direct manipulation is broken
Why the IoT asks consumers to think like programmers, and the UX
challenges this creates
Claire Rowland / @clurr
O’Reilly Design Conference
21st January 2016

2. “This is more than a UX book; it covers all of the critical design and
technology issues around making great connected products.”
DAVID ROSE, ENCHANTED OBJECTS
“As a grizzled veteran of several campaigns within the matter-battle
of the Internet ofThings, I was pleasantly surprised to find the
number of times this book made me pause, think, and rethink my
own work (and that of others).A very valuable addition to the canon
of design thinking in this emerging area.”
MATT JONES, GOOGLE
“Connected products are a unique beast. Designing this type of
product well takes lots of iterations & user testing.This book takes
you through the paces.We keep one on the shelf at Highway1 for
our IoT companies.”
BRADY FORREST, HIGHWAY1.IO

3. “The IoT breaks direct
manipulation. Using
connected devices is
conceptually more like
programming.”
Alan Blackwell
Professor of Interdisciplinary Design,
Computer Laboratory,
University of Cambridge
Very roughly paraphrased

4. “The IoT breaks direct manipulation.
Using connected devices is conceptually more like
programming.”

5. Direct manipulation
“A human–computer interaction style which involves continuous representation of objects
of interest and rapid, reversible, and incremental actions and feedback.”
Ben Shneiderman
“The future of interactive systems and the emergence of direct manipulation". Behaviour & Information Technology

6. Why direct manipulation works
- Visibility. By looking, the user can tell the state of the
device/system and the alternatives for action.
- A good conceptual model. The designer provides
a good conceptual model for the user, with consistency
in the presentation of operations and results and a
coherent, consistent system image.
- Good mappings. It is possible to determine the
relationships between actions and results, between
controls and their effects, and between the system state
and what is visible.
- Feedback. The user receives full and continuous
feedback about the results of actions.
Chapter 2,The Design of Everyday Things

7. A key condition of direct manipulation is that the user’s
actions and the results happen
at the same time, in the same situation

8. Remote control
Image: British Gas

9. Automation
Scheduling things to happen in future, like turning on lights to welcome you home, or turning
off unnecessary appliances to save energy overnight.
Image:; Martin Cathrae via Flickr

10. Interoperable, coordinating products
Video: shot atTarget Open House
Things in your world will
work together to meet and
maybe even anticipate your
needs.
A sleepsuit that detects
your baby waking up could
trigger a morning routine.
An “intelligent” security
system could draw on
information from your
calendar, to detect possible
intrusions.

11. All these scenarios break direct manipulation because
they displace user actions from system responses

12. Remote control:
displacement in space
You might:
- Set the security alarm when there’s still
someone at home
- Turn on lights when they’re trying to sleep
- Turn the heating off when someone needs
it
- …or turn on an electrical appliance when
there’s a gas leak in the home
Image:Audetat via WikiCommons

13. Risks of displacement in
space
- The action may be inappropriate or
unsafe at that time (or just annoying to
someone else)
- Users may not see the consequences of
their actions
- Users may not realise that the action
they thought they took didn’t work as
planned
Images: https://tiare75.wordpress.com

14. Is that 7pm where I am
now, or at home?

15. Automation:
displacement in time
- That “unnecessary” appliance that
gets turned off might be the
tumble dryer running overnight to
ensure a shirt is ready for an
important meeting.
- Your alarm goes off because your
calendar says you’re not ‘supposed’
to be home and infers you might
be an intruder.
- Your baby wakes up super early
and you don’t want to start the
day, you want to GO BACKTO
SLEEP
Google Image search for baby waking early

16. Risks of displacement in
time
- Users need to anticipate their own
future needs: which humans are not very
good at.
- Users may forget that they have set up
an action, and then be negatively
affected by the execution of that action.
- They need to anticipate all the changing
conditions to which the system might
have to respond.
http://www.independent.co.uk/sport/rugby/rugby-union/international/barbarians-vs-samoa-
interrupted-by-sprinklers-as-fans-criticise-no-beer-lack-of-wi-fi-and-poor-10478187.html

17. Flexible, multi-purpose hardware
Automated security lighting schedule?
Monitoring energy consumption of an appliance?
Remote control of your kid’s Xbox?
Image: Belkin
Functionality can be modified in a remote software layer

18. Smart plug that ran an automated schedule
for a lamp is moved to hair straighteners…
and is now turning those on and off when
you’re not at home
Image:; London Fire Brigade
Flexible, multi-purpose
hardware: displacement
in function

19. Risk of displacement of
function
- The behaviours of the device that were
appropriate for its original use are ported
over to a new use, and are now actively
dangerous
Image: Shropshire Insurance

20. But hey, that’s why we
need to make things
smarter, isn’t it?

21. - You could have a gas sensor that alerts
you and turns off the electricity if it
detects a leak.
- You could come up with a way of
temporarily excluding the tumble dryer
from your energy saving program.
- You could program the smart plug to
identify changes in energy consumption
that might indicate that it’s being used
with a different device, and disable the
schedule.
- You could disable the alarm with two
factor voice authentication: not just
keycode or phone, but voice.
- You could have that baby’s room macro
run only after 6am.
Image:; meetangee.com

22. Indeed you could.
But it’s very hard to
mitigate every possible
problem.
Number of edge
cases you’d have
to anticipate
Few
Tons
Discrete product Many interconnecting
products

23. “…the cumulative complexity of a bunch of simple
things—regardless of how delightful, simple and
desirable they may be—will soon exceed the ability
of humans to cope.”
Bill Buxton
Principal researcher, Microsoft Research
http://www.fastcodesign.com/3054433/design-moves/the-most-
important-design-jobs-of-the-future

24. In Norman’s terms, the design upshot can be:
- Poor visibility. It’s not always possible to tell, by looking, the state of the device or
the actions that are possible with it.
- A complex conceptual model.The conceptual model of a connected product
is way more complex. Need to understand parts, what each does and how they
interrelate. Separation of operations and results can make it harder to identify
consistencies.
- Obscured mappings. It is harder to determine the relationships between actions
and results, between controls and their effects, and between the system state and
what is visible.
- Poor feedback.The user may get only partial, delayed, incorrect or intermittent
feedback about the results of actions.

25. “The IoT breaks direct manipulation.
Using connected devices is conceptually more like
programming.”

26. Since when did turning lights on and off count as
programming?
Images Keil C51 C Compiler (Phaedrus Systems), Philips

27. Let’s break down a home automation task…

28. I want a nice green lawn
Understand and frame your own problem.
= Requirements definition
Image:Adam Kerfoot Robverts via Flickr

29. “I need a controller, and some sprinklers, and I need
to plan around weather, water use restrictions, and
social events”.
= Specification
Image: Blossom

30. “Now… how do I get this thing to do what I want?”
Image: Rachio Iro
= Implementation

31. “Oops.What went wrong?”
=Testing and debugging
Image: http://blogs.ifas.ufl.edu/pinellasco

32. This is software engineering!
Users must:
•Anticipate their own future needs and plan a solution
•Configure the system to meet these needs using an
abstract representation
•Deal with unforeseen conditions that may make the
system behave in unexpected ways
•All the while… considering future conditions that might
cause unexpected effects or conflicts
Requirements definition
Specification
Implementation
Testing/debugging

33. You say that like it’s dirty.

34. Empowering people to program is great

35. But it’s a lot of work for a consumer product
Product Tool
In areas where they don’t have expert knowledge or are short on time, consumers
tend to look for products which come preconfigured to solve a specific problem, rather
than multi-purpose tools which require the user to define and solve their own problem.

36. It’s a time and
attention issue
Image: XKCD

37. OK.
But that still doesn’t explain how using a simple
remote control is like programming

38. Users need to engage with the system model
Internet
Local WiFi

39. Programming can be hard, and painful, however
good at it you are

40. Experts *expect* to
make mistakes
(and are often paid to do so)
I expected it wouldn’t be perfect
the first time around and there is
always an exception.You learn about
them after you start using them.
Expert home automation user
(and IT security professional)

41. Image: Zach Luz (zachluz.com)
…users don’t
(and are never paid to do so)
- Things not working isn’t part of the
normal path to success for a consumer
product… but this will happen
- It’s especially painful at home: which is
supposed to be our haven from the
world
- System designers need to do all they can
to mitigate potential damage

42. The logic of even apparently quite simple things can
be complex… and inconsistent
if (time == 1800) then set light =on
if (time == 2200) then set light =off
while (mode ==scheduled) then
{
while (time >=1800 && time <=2159) then set target temperature =72F
while (time >=2200 && time >=0659) then set target temperature =65F
while (time >=0700 && time >=0859) then set target temperature =72F
while (time >=0900 && time >=1759) then set target temperature =58F
}

43. What do we do about it?
Or, how can we make controlling complex systems easier?

44. Home Maestro
http://research.microsoft.com/apps/pubs/
default.aspx?id=70594
This is not (just) about
new types of
programming interface
They might help with the implementation
part of the problem.
But they don’t help with the problems of
predicting your needs, planning and
debugging, forming a conceptual model or
knowing what’s happening at a remote
location.

45. Images: Lowes
Better feedback: was my command executed or not?

46. Better visibility/feedback: remote context
- Am I about to do something that
might be unwise?
- Has something happened that might
not be what I want?
Carrier 12:00 PM
!! Jen is at home !!
ARM SYSTEM
Home security alarm

47. Better visibility: why did that thing happen/not happen?
- “Why didn’t the dishwasher run
this morning?” (perhaps the smart
electricity grid turned off the supply to that
nonessential device at a time of peak
demand)
- “Why is the hall light flashing on
and off?” (perhaps it is receiving conflicting
instructions from security and energy saving
apps)
http://www.cs.cmu.edu/~NatProg/whyline.html

48. Error prevention
- Flag up potential consequences:
sandbox simulations to check for
rule conflicts
- Try to prevent the user from doing
dangerous things
Your new lighting rule
conflicts with your
security rule
I’m pretty sure that’s no
longer a lamp I’m
powering

49. What about autonomous systems?
Scott Jenson: Home automation is an easyhard problem
https://jenson.org/easyhard/
…common sense is hard to get right

50. What about us?
Will we adapt to a
programming-like world?

51. [Ben Evans]
My grandfather could probably have told you how many electric motors he owned.There
was one in the car, one in the fridge, one in his drill and so on.
My father, when I was a child, might have struggled to list all the motors he owned (how
many, exactly, are in a car?) but could have told you how many devices were in the house
that had a chip in.
Today, I have no idea how many devices I own with a chip, but I could tell you how many
have a network connection.And I doubt my children will know that, in their turn.
Benedict Evans
Andreessen Horowitz
http://ben-evans.com/benedictevans/2014/5/26/the-internet-of-things

52. If you’ve enjoyed this, the book covers:
- How the technology of IoT affects UX
- Product and design strategy for connected devices
- Industrial design
- Interface and interaction design for embedded devices
- Cross-device interactions and interusability
- Interoperability
- Responsible IoT design
- Designing with data
- Prototyping and user research methods for connected
products
www.designingconnectedproducts.com
@clurr | claire@clairerowland.com