People use smartphones in daily activities for accessing and storing information in various situations. In this paper, we present a work in progress for detecting and automating some of these activities. To explore the possible patterns we developed an experimental application to detect daily tasks used by smartphones and analyzed it to provide suggestions for “routines”. We conducted a two-week user study with 10 users to evaluate the approach. During the study the application logged the usage patterns, sent information to the server where it was analysed and clustered. The participants could also automate their smartphone tasks using the analysed data. The findings suggest that people would be willing to automatize tasks given that the approach gives flexibility and expressiveness without too much information overload. Future work includes refining the algorithms based on the gathered real-life data and modifying the interaction design to approach the challenges found with the initial study.

1. RoutineMaker: Towards End-User Automation of Daily Routines Using
Smartphones
Ville Antila, Jussi Polet, Arttu Lämsä, Jussi Liikka
Context-Awareness and Service Interaction
VTT Technical Research Centre of Finland
Oulu, Finland
{ville.antila, jussi.polet, arttu.lamsa, jussi.liikka}@vtt.fi
Abstract — People use smartphones in daily activities for the context or situation of the user, maybe even better than
accessing and storing information in various situations. In this more traditional and quantifiable sensors of context can.
paper, we present a work in progress for detecting and In this paper, we study the possibilities of detecting and
automating some of these activities. To explore the possible automating smartphone usage routines. With a routine we
patterns we developed an experimental application to detect mean an association of a location, used application and the
daily tasks used by smartphones and analyzed it to provide time of day. To reveal some of these somewhat hidden
suggestions for “routines”. We conducted a two-week user patterns, we developed an application to detect the day-to-
study with 10 users to evaluate the approach. During the study day smartphone use by logging the application usage and
the application logged the usage patterns, sent information to
locations and clustering them to identifiable patterns. We
the server where it was analysed and clustered. The
participants could also automate their smartphone tasks using
also implemented a functionality to automate these patterns
the analysed data. The findings suggest that people would be using the application. One reason for this functionality was
willing to automatize tasks given that the approach gives to find out whether the users could actually detect some
flexibility and expressiveness without too much information routine-like behaviour from their smartphone usage patterns;
overload. Future work includes refining the algorithms based which would then help us to evaluate our approach
on the gathered real-life data and modifying the interaction qualitatively.
design to approach the challenges found with the initial study.
II. RELATED WORK
Keywords - Context-awareness; Routine detection; Sensing; The idea of extracting usage patterns and routines from
Smartphones; Task automation; smartphone usage data is not unique or novel as such. There
has been a body of research exploring different quantitative
I. INTRODUCTION methods to mine patterns of human activities from large
datasets. Eagle and Pentland demonstrate the ability to use
Smartphones are becoming ubiquitous and ever more
mobile devices to recognise social patterns, identify
important for the daily activities of their users. The multitude
significant locations, and model organisational rhythms [4].
of smartphone applications, dedicated to help in daily tasks,
Farrahi and Gatica-Perez suggest that human interaction
are used almost everywhere at any time. Smartphones and
data, or human proximity, obtained by mobile phone
their applications, serving as pocket PCs and extending our
Bluetooth sensor data, can be integrated with human location
desktop experience, are becoming so ubiquitous part of our
data, obtained by mobile cell tower connections, to mine
ways to store and access information that some of the tasks
meaningful details about human activities from large and
we perform with them have become daily routines. Examples
noisy datasets [6]. They also present a framework to classify
of routine-like behaviour can include checking e-mail in the
people’s daily routines (defined by day type and by group
morning, reading the news or listening to music while
affiliation type) from the data [7]. Similarly Verkasalo
commuting, searching local information, navigating or
illustrates the relationships between common locations, such
checking-in to places to assess and comment our on-the-go
as office or home, to the usage patterns of different
experiences. People also use smartphones to complement
applications [10]. In our work we are concentrating on real-
other daily activities or routines such as watching TV,
time analysis and presentation of routines of an individual
reading newspaper and going to the grocery store [8].
user rather than modelling the group behaviour. We are also
On the other hand, the latest consumer studies indicate
looking into qualitatively evaluating the found patterns by
that the emerging user patterns could be more application-
the user (by the act of saving or modifying the routine).
specific than they are device-specific [5]. The routine of
Chittaranjan et al. investigate the relationship between
“checking Facebook in the evening from bed” could be done
behavioural characteristics derived from rich smartphone
either with a smartphone, laptop or a tablet device. The
data and self-reported personality traits [1]. The data stems
action or behaviour is often associated to a specific service or
from smartphones of a set of 83 individuals collected over a
application it is done with more than the device mediating
continuous period of 8 months. From the analysis, they show
the experience. Therefore we can hypothesize that the usage
that aggregated features obtained from smartphone usage
of a specific application can also indicate something about
data can be indicators of the Big-Five personality traits.

2. Additionally, they present an automatic method to infer the automated routine out of it. The saved routine is then sent to
personality type of a user based on cell phone usage with up the server as well for persistent storage and further analysis.
to 75.9% accuracy. This work gives an interesting insight
into how the collected behavioural data can be related to B. Mobile Application
known personality traits, and as a concept could be applied The RoutineMaker mobile application visualises the
in our research as well in the future. detected routines (see Figure 1) by showing the location
In addition to detecting the routines using smartphones, clusters on a map view as well as on a list view. The
there has been research on how to present it to the user for a MapView shows the location cluster markers, by which
potential user action. Dearman et al. present an approach to tapping the user can see a preview list of the most used
present information to the user based on the location and applications in that cluster. The user can also switch from the
knowledge of the task. Examples include location-based task MapView to the ListView, which shows an in-depth list of
notifications and support for opportunistically suggesting the location clusters. In the ListView, the user can select
places for certain activities on-the-go [2, 3]. While these desired applications to be launched at specific times and save
studies have similar goals than our approach, the intended the sequence as a routine. The RoutineMaker mobile
usage situations are somewhat different; nevertheless we application checks frequently, if there are any routines to run
think that the application presented in this paper could and if so, it checks whether the current location and time is
benefit from introducing some form of serendipitous or associated with any routines. Should there be a match; the
opportunistic presentation of data to the user regarding the specified routine is run automatically.
routines.
We also acknowledge in our work that the breadth of
analysis done with the data can also be potentially misused.
Shilton discusses the privacy of collecting multi-dimensional
sensor data from mobile phones [9]. As by using
smartphones it is possible to gather an extensive set of
information about people’s locations, habits and routines,
even personality traits, it might be that smartphones at the
extreme could be the most widespread embedded
surveillance tools in the history. The trade-off for the user is
between the perceived benefit and privacy concerns, and we
see that this trade-off should be balanced by the user via her
actions using the system (explicitly sharing what is needed
and wanted to be shared).
III. ROUTINEMAKER APPLICATION
In this section we present the developed application for
detecting and automating daily routines with smartphones.
The application logs daily smartphone usage data (locations, Figure 1. Cluster overview shown in the MapView and
time and used applications) and tries to detect patterns, such cluster details shown in the ListView
as a sequence of applications used or tasks done on a certain C. Server
time at a certain location. Once the possible routines are
The server-side application is responsible for creating the
detected, the application displays them to the user. The user
application-location clusters from the logged data received
can accept and create a “routine” from the suggested
from the client devices. The algorithm is split into two main
patterns or modify the suggested pattern and then save it.
phases: geographical and application clustering. The steps
The user can also name the routines in similar way than one
are illustrated in Figure 2.
would do with ordinary applications (e.g. “going to sleep”-
First step of the process is the geographical clustering
routine, “going to movies”-routine, “going to work”-
which filters out the most significant locations from the data
routine).
(visited or stayed most often). After the geographical
A. Software Design and Implementation clustering is done, an application table is generated, where
The prototype consists of a mobile application, which each column represents a five-minute time slot in a day and a
collects usage data and presents the processed usage data to row is generated for each application. Then the cluster
the user and a server-side application, which performs the samples are gone through and the value of the table element
data processing. The mobile application gathers usage data representing the time-application combination of the sample
(location and applications used) from the device. This data is is increased by one. After this, the whole table is normalised
sent to the server and analysed to find location clusters and by dividing it by the maximum element of the table. A usage
used applications in those clusters. The mobile client table, containing Boolean values, is generated from the
presents this analysed data to the user. If the user notices application table. The usage table is the same size as the
helpful or useful routines from the data she can create an application table and the elements contain value true, if the
application table value in this element is greater than a

3. threshold value, otherwise the elements contain value false. Table 3 Research questions
This is followed by applying a smoothing filter to the usage ID Question
table. This removes false slots that are located in between RQ-1 Is it possible to extract routines or tasks from the historical
two true elements in the usage table. usage data?
RQ-2 Were the extracted and suggested routines helpful?
RQ-3 [Following from the RQ-2] Could they be useful?
Geographic clustering Application clustering RQ-4 [Following from the RQ-2] Did they reveal any other possibly
interesting or important information?
Add samples to clusters Generate table of active
applications ordered by time
A. Participants
Filter out clusters with not Normalize table and get We recruited ten participants from three countries using
enough samples application usage times to e-mail lists. There were nine male participants and one
usage table
female. The participants had to be active smartphone users.
Combine clusters close to The participants also had to have suitable mobile phones
each other Apply smoothing filter to supported by the application (Android v2.2 or higher). The
usage table
participants were in the age range of 27 to 33 years with
Filter out samples far away average age of 29.7 and were very active smartphone users,
from cluster centers Get application launch times as 62% of them used smartphone applications a couple of
from usage table
times a day and 25% used them a couple of times in an hour.
Figure 2. Algorithm structure (repeated for each user) B. Findings
In this section, we provide a brief analysis of the gathered
The application launch times are then read from the data. The sources for the gathered data are the initial
usage table. Always when an element containing the value questionnaire, the logged data from the user study, the post
true is found proceeded by false; an application launch time questionnaire and open ended questions the users were asked
is detected. Table 1 and Table 2 contain an example of the in the end of the study.
application and usage tables. The generated application table
1) Perceived usefulness of routine detection and the
is shown in the Table 1. The usage table shown in the Table
2 is obtained by using a threshold value of 0.7. In this RoutineMaker application
example, two launch times are detected; 13:05 for “Music First, we asked how useful the participants rated
player” and 13:20 for “E-mail”. detecting their smartphone usage routines. The results show
that this was considered as useful (avg. 3.7, sd. 0.8, on a
Table 1 Application table scale from 1 to 5). We also asked how useful they perceived
Application Time
the RoutineMaker application as such. The results showed
13:00 13:05 13:10 13:15 13:20 13:25 13:30 that the approach was not perceived as very useful (avg. 2.1,
Music player 0 0.8 0.74 0.8 0.4 0 0 sd. 0.9). The reason for this was visible in the comments:
Web browser 0 0 0 0 0 0 0 First of all, the application could only automatically launch
E-mail 0 0 0 0 0.9 1 0 applications when they were at a certain location at a certain
Notebook 0 0 0 0.3 0.1 0.2 0 time. What the users wanted was even more automatic
behaviour, such as performing a certain task on its own
Table 2 Usage table without any user intervention. With the current design, such
Application Time
13:00 13:05 13:10 13:15 13:20 13:25 13:30
elaborate tasks were impossible to make with the application.
Music player false true true true false false false This lowered the perceived usefulness. Nevertheless, these
Web browser false false false false false false false comments give good insight into developing the application
E-mail false false false false true true false further.
Notebook false false false false false false false 2) Understanding the important factors of smartphone
routine detection
IV. USER STUDY To get a better insight into the design space of
The RoutineMaker application was evaluated with ten smartphone routine detection, we asked the participants to
users, who used the application for two weeks. During the rate 4 different factors or parameters of the routine detection
two weeks, the application logged the routines of the user, on a scale from 1 to 5. These parameters were: quality of
sent this information to the server, where it was analysed and detected patterns, amount of detected patterns, resolution of
clustered. The participants could automate their smartphone detected patterns and the accuracy of detection. Based on the
tasks using the analysed data. In the study, we included a results, the most important factor was accuracy (avg. 4.3, sd.
start and end questionnaires and a set of open-ended 0.76), while the amount of detected patterns was rated as
questions to probe the participants about their needs and least important (avg. 2.9, sd. 0.9). Resolution and quality
experiences related to the application concept and the actual were rated as somewhat important factors (avg. 3.6, sd. 0.79
usage of the prototype application. The research questions and avg. 3.3, sd. 1.1). The standard deviation was large in
for the user study are listed below in Table 3 Research quality ratings; taking a closer look at the results it seems
questions). that some participants did think that quality was important

4. whereas some did not rate it as important. It is possible that still offering suggestions based on the detected behaviour,
quality as a measurement was not very well understood in we can enable an easy and fast interface for users to
this context, or that it was already incorporated in the other customise and automate their routine-like behaviour without
ratings. limiting only to the specific smartphone tasks.
We also asked how well the RoutineMaker application The lessons learned from the application development
performed regarding the selected factors (quality, amount, and the user study include that the amount of detected
resolution and accuracy). In general, the application clusters (potential routines) can be quite high, therefore
performance was in line with the importance of the factors. leaving the selection and creation of routines more to the
The accuracy of routine detections was rated good (avg. 3.5, user. Nevertheless the suggestions should include only
sd. 0.84), as well as the resolution of the detected routines relevant applications, which are detected usually during the
(avg. 3.25, sd. 1.17). These were rated as most important same times during the day, in a routine-like manner.
factors, so we can conclude that some of the parameters The future work consists in developing the application
selected for the routine detection algorithms were and algorithms further using the data gathered from the
corresponding to what the participants thought as important study. In some cases the algorithm for detecting the routines
or useful. was “too adaptive” and some clusters were removed if the
The amount of detected routines was rated the worst of user had an unordinary day during the week. We are seeking
these factors (avg. 2.25, sd. 1.05). This can be due to a to tweak the threshold for the adaption and include better
relatively large number of false positive detections of fitness values for the detected, possible routines, and
applications due to the features of the underlying OS weighting them in the algorithm enabling the process to learn
(Android), which opportunistically leaves applications which kind of application sequences are perceived as useful
running in the background to increase user experience (such routines. We are also looking into doing more user studies
as the response times of applications). This issue can be with larger group of participants learning more about the
fixed by filtering out processes which the user is not actively user behaviours and surveying the routines people currently
using. Nevertheless, we can also hypothesise that leaving have while using smartphones.
some of these applications to be selectable can create certain
serendipity in creating the routines and allows users to create REFERENCES
routines that necessarily were not detected as such, but could [1] Chittaranjan, G., Blom, J. & Gatica-Perez, D., Who’s Who with Big-
be useful in the future (some of the processes are useful for Five: Analyzing and Classifying Personality Traits with Smartphones.
tasks even when they are not active on the UI). Proceedings of the International Symposium of Wearable Computing
(ISWC2011), IEEE Computer Society, 2011.
V. DISCUSSION AND FUTURE WORK [2] Dearman, D., Sohn, T. & Truong, K. N. Opportunities Exist:
Continuous Discovery of Places to Perform Activities. Proceedings of
After the user study, we asked the participants whether the 2011 Annual Conference on Human Factors in Computing
the application usage revealed any surprising facts about the Systems, ACM, 2011.
people’s smartphone usage. Many participants answered that [3] Dearman, D. & Truong, K. N., Identifying the Activities Supported
the application did reveal routines, but these were largely by Locations with Community-authored Content, Proceedings of the
12th ACM International Conference on Ubiquitous Computing
known and thus not really surprising as such. In some cases (UbiComp'10), ACM, 2010.
the participants were surprised of the prevalence of these [4] Eagle, N. & Pentland, A., Reality Mining: Sensing Complex Social
routines. The detected and “accepted” routines were, for Systems, Personal and Ubiquitous Computing, Vol. 10, No. 4, pp.
example, checking e-mail in the morning at home, or 255-268, 2006.
checking-in to Foursquare at certain locations at certain [5] Ericsson Inc., From Apps To Everyday Situations - An Ericsson
(rather fixed) times. Some routines included using sports Consumer Insight Summary, 2011, Available:
tracking when going for a jog or bicycling. Some routines http://www.ericsson.com/res/docs/2011/silicon_valley_brochure_lette
r.pdf.
included a set of detected applications, like alarm clock and
email client in a sequence. [6] Farrahi, K. & Gatica-Perez, D., Probabilistic Mining of Socio-
Geographic Routines from Mobile Phone Data, Selected Topics in
We can conclude based on the study that we found Signal Processing, IEEE Journal, Vol. 4, No. 4, pp. 746-755, 2010.
distinct usage patterns which can be potentially automated. [7] Farrahi, K. & Gatica-Perez, D., Daily Routine Classification from
In addition, the users were interested in creating automated Mobile Phone Data, Machine Learning for Multimodal Interaction,
routines based on their smartphone usage behaviour, but in 2008.
many cases these routines were more elaborate and complex [8] Google Inc., The Mobile Movement - Understanding Smartphone
than our application could offer. Examples include time and Users, 2011, (last updated on 26th April 2011). Available:
location-based triggers with variable sequences, such as http://googlemobileads.blogspot.com/2011/04/smartphone-user-
study-shows-mobile.html.
“change my profile to silent when entering certain location
[9] Shilton, K., Four Billion Little Brothers? Privacy, Mobile Phones and
between certain times”. Going further, we can envision Ubiquitous Data Collection, Communications of the ACM, Vol. 52,
location and time dependent notifications like to-dos or No. 11, pp. 48-53, 2009.
calendar entries or tasks that could be notified when the [10] Verkasalo, H., Contextual Patterns in Mobile Service Usage.
device senses idleness (suggesting upcoming tasks while Personal and Ubiquitous Computing, Vol. 13, No. 5, pp. 331-342,
sitting in a bus for example). We argue that by giving the 2009.
user control over how these tasks could be performed, while