Invited talk at Human Media Interaction unit at University of Twente

1. From Context-Awareness to
Human Behavior Patterns
Detection of Daily Routines using Smartphones
Ville Antila (Research Scientist, M.Sc.)
VTT Technical Research Centre of Finland, Oulu, Finland
Philips Research, Eindhoven, The Netherlands (Visiting researcher)

2. Background – Smarcos project
• Smarcos creates solutions to allow
devices and services to exchange
context information, user actions,
and semantic data
• One important part of the work has
been to investigate the practical
usage of context information and to
develop models that can be
dynamic and adaptive as well as
applicable to different applications
• www.smarcos-project.eu

3. Outline of the talk
• Introduction
• From context logging to routine detection
• Continuos, low-power “life-logging”
• Interpreting the data (what’s the meaning of it)
• Using domain knowledge to reason about what we don’t know
• Example applications
• Discussion and summary
• Video?

4. Introduction - context awareness
• Idea that computers can both sense and react to their environment
• “any information that can be used to characterize the situation of an entity” [Dey et al., 2001]
• Human factors: information about the user, social environment, user’s task
• Physical factors: Location (absolute, relative, co-location), infrastructure, physical conditions
• Context aware systems should be able to gather context (sense the situation),
abstract and understand it and adapt application behavior based on the context
• Some classical use cases:
• adapt user interfaces
• tailor application-relevant data (e.g. filtering information)
• increase precision of information retrieval
• make the user interaction implicit
• discover services & build smart environments

5. Introduction - context awareness
• Smartphone is the epitome of a sensing platform
for context-awareness
• Personal and mobile (almost always with the user)
• Comes with a lot of in-built sensors and
communication capabilities
• Used everywhere, for multiple tasks (on-the-go)

6. Introduction - challenges & opportunities
• The notion of ‘context’ can not be objectively defined (a prior) by settings,
actions and actors
• Rather, context is the meaning that the actions and actors acquire at any
given time from the subjective perspective [Mancini et al., 2009]
• Context information can have multiple purposes (from user’s perspective)
• “Declaring one’s position is perhaps as much about deixis (pointing at and referencing features of the
environment) as it is about telling someone exactly where you are” [Benford et al., 2004]
• When going from individuals to larger groups of people it’s possible to extract
patterns from the data [Eagle and Pentland, 2006]
• social patterns (e.g. urban heat maps)
• identification significant locations
• model organizational rhythms

7. From Context Logging to Routine Detection
Applications • Example applications...
Support for decision • Use the gathered knowledge to form
decisions about system behavior in
making different contexts/situations
Domain • Infer new knowledge from information
Reasoning
knowledge
• Transform raw data into information
Context Interpreter about the user’s context
Data Layer • Capture raw data from device sensors

8. Data Layer
Continuous Life Logging
• Context-awareness
• Using smartphones as sensors for human
activities (e.g. important locations, mobility
patterns)
• Low-power context logging software
• Semantic location detection using cell-id
(low power, always available)
• Device usage detection (algorithms for
mining location relative to smartphone
application usage)
• Lo-fi physical activity detection (e.g. is the
user moving currently)
• Scanning Bluetooth snapshots to
determine indoor environment (e.g. is the
user at his office desk)

9. Context Components
For real-time user behavior detection
• Collection of software components for
enabling
• continuous context logging
• development of context-based adaptation for
variety of applications
• Implementations available for several
platforms (some of which are becoming
obsolete)
• Android, Symbian, Maemo/Meego/Linux,
BlackBerry

10. Context
Interpreter
Interpreting the data...
• Estimation of life patterns
such as the semantic location
of the user (e.g. “home”,
“office”)
• Detection of device usage in
different locations
• Detection of physical activity
in different situations
• Detection of changes in
routines

11. Context Capture
Context-based awareness cues in information sharing
• We explored the usage of contextual information
cues in informal information sharing
• The study focused on practices of ‘abstraction’
when publicly sharing contextual information
• Field test for our backend

12. Smartphones are used almost everywhere...
Moreover there is an “app” for almost anything...
An opportunity (to use smartphone
apps as sensors for situations...)?
Image sources:
http://adage.com/article/digital/placing-ads-underestimate-mobile/230853/
http://www.resultrix.com/blog/index.php/tag/tablet-marketing
http://www.google.com/googleblogs/pdfs/mobile_understanding_smartphone_users.pdf

13. Routine Maker
End-user automation of smartphone routines
• An approach to detect day-to-day
‘routines’ by logging the
smartphone application usage and
locations where they are used
• Analysis of logged usage data into
identifiable patterns (clustering
based on location and time of use)
• Implemented an experimental
smartphone application with a
functionality to create
automated ‘tasks’ out of the
identified patterns
• Conducted a two-week user
study to analyze the approach
and to gather user feedback

14. Domain knowledge Reasoning
Extending the knowledge...
• By modeling domain knowledge we can reason about the consequences of what
the derived context information means in the particular application scenario
• For example, if know (to a certain degree of accuracy) that the user is cycling,
then we can reason that:
• available devices are mobile devices (phone & activity monitor)
• availability for receiving messages is low

15. Where would this information be useful?
• Determine the devices that surround the user
• e.g. at work, the user has access to his personal computer
• Time and adapt system feedback based on the situation
• e.g. time-shift notification to where user is more receptive
• Log important events and use those to automate tasks
• e.g. migrating task sessions automatically between personal devices

16. Applications
Example application:
Context-adaptive Feedback
• Goal: increase effectiveness and
decrease interruptions
• How: adaptive selection of device,
modality and timing of feedback
• Example of adaptive feedback
delivery:
• IF the situation is suitable, THEN
send the message (as it is)
• IF the situation is not suitable AND
the message is not urgent, it should
be time-shifted
• IF the situation is not suitable AND
the message is urgent, then the
content should be adapted to the
situation

17. Applications
Example application:
Context-based User Interface Migration
• Migration concept
• Interface moving from source device to target device
• Interaction state persistence
• Interaction continuity

18. Applications
Example application:
Context-based User Interface Migration
• Taking advantage of the known properties of the environment in any given time (e.g.
Bluetooth, GPS, device stability and orientation) we can automate tasks such as UI
migration triggering

19. Discussion
• Just like the smartphone sensor APIs have matured (GPS, acc, gyro,
proximity), the basic context abstractions will also be served as OS-
level services in the future (e.g. walking/running/still, home/office/
school/on-the-go)
• Better optimizations regarding battery, CPU and memory usage etc. (resolves the
iOS background processing challenge)
• Cross application usage
• Fusion with other available data on the device and “in the cloud”
• Going further we can also foresee taking inputs from the environment
(e.g. WSNs) as well as “negotiating with” other smart devices while
trying to reach a better approximation of the situation

20. Discussion - challenges
• Quality/accuracy of detection (have to be started off with simple
cases), provenance of quality measures into the application level
• User interaction/awareness of application behavior
• Designing for applications that adapt to situations... Patterns,
guidelines, best practices?
• Prototyping context-adaptive applications (from early interactive
prototypes to functional prototypes)
• Testing
• Functional testing, performance/quality testing
• User testing (‘in the wild’)

22. From Context-Awareness to Human Behavior Patterns
Detection of Daily Routines Using Smartphones
Thank you!
Questions?
Ville Antila
ville.antila@vtt.fi