A presentation given by Erik Swan, CTO/Co-Founder of Splunk and Michael Wilde, Splunk NInja at the SXSW Interactive 2012 Conference on March 11, 2011

1. Erik Swan, Michael Wilde
Big Data for
Everyman

2. Hi... We work at Splunk.

3. We stare at data all day.

5. WTF is Big Data?!

6. larger than small data?

7. smaller than giant data?

8. some cool sauce for DBAs?

9. Aaaahhh, no.

10. a simple way to describe a
massive problem
*or opportunity depending on your p.o.v.

11. Big data comes out of machines
GPS,
RFID,
Hypervisor,
Web Servers,
Email, Messaging
Clickstreams, Mobile,
Telephony, IVR, Databases,
Sensors, Telematics, Storage,
Servers, Security Devices, Desktops
Volume | Velocity | Variety | Variability

12. Big data comes out of machines
Machine-generated data is one of the
GPS,
fastest growing, most complex RFID,
and most valuable segments of big data Hypervisor,
Web Servers,
Email, Messaging
Clickstreams, Mobile,
Telephony, IVR, Databases,
Sensors, Telematics, Storage,
Servers, Security Devices, Desktops
Volume | Velocity | Variety | Variability

14. no, not us
we’re just
nice guys
who want
show you
cool stuff

15. building a service?
you are a producer and
consumer of data
using an app?

16. Seth Rabinowitz James Rodmell
CEO CTO
Location-­‐Based
Messaging
and
Intelligence
For
Your
App
and
Your
Customers

17. DATE/TIME
Data! Good!
DEVICE ID
2011-11-06 11:57:31,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.75496,-73.963853,60
2011-11-06 12:17:32,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.755001,-73.963886,70
2011-11-06 12:37:33,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754982,-73.963849,75
LAT/LONG
2011-11-06 12:57:34,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754984,-73.963883,85
2011-11-06 13:17:35,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754941,-73.9639,90
2011-11-06 13:37:36,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754948,-73.963874,90
2011-11-06 13:57:37,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754931,-73.963892,95
BATTERY STRENGTH
2011-11-06 14:17:38,50,00027d27-ae02-627d-a79a-fa0004d3a347,40.755232,-73.963522,100
2011-11-06 14:37:33,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754979,-73.9639,100

18. show them something
cool already!

19. Oh, real quick. Did you check in
or tweet #splunk #sxsw
...please

20. All this data can be pretty cool
and empowering

21. except one little
PROBLEM Text

22. alot of it looks like this

23. 0,1
13/Apr/2011 08:52:53,Info,Teardown,ASA-session-6-302014,TCP,
192.168.2.16,192.168.1.6,(empty),(empty),1100,43025,43025_tcp,
(empty),0,1
13/Apr/2011 08:52:55,Info,Teardown,ASA-session-6-302014,TCP,
192.168.2.75,192.168.1.6,(empty),(empty),1048,135,epmap,(empty),
0,1
13/Apr/2011 08:52:55,Info,Teardown,ASA-session-6-302014,TCP,
192.168.2.75,192.168.1.6,(empty),(empty),1049,43025,43025_tcp,
(empty),0,1
13/Apr/2011 08:52:55,Info,Teardown,ASA-session-6-302014,TCP,
192.168.2.75,192.168.1.6,(empty),(empty),1051,135,epmap,(empty),
0,1
13/Apr/2011 08:52:55,Info,Teardown,ASA-session-6-302014,TCP,
192.168.2.75,192.168.1.6,(empty),(empty),1052,43025,43025_tcp,
(empty),0,1
13/Apr/2011 08:52:55,Info,Teardown,ASA-session-6-302014,TCP,
192.168.2.64,192.168.1.6,(empty),(empty),1694,135,epmap,(empty),

24. and we’re expected to talk to
it like this

25. select (select max(answer.answer) from answer where answer.member_id in (
select member_id from team_members where project_id in ( select project_id
from project where Business_stream='Upstream' and stage='Appraise' and
project_id in (select project_id from projectextra where subteam<>1 ) ) ) and
answer.page_id=page.page_id) as thinl, (select max(avgscore) from task_project
where task_project.project_id not in (select project_id from projectextra
where subteam=1 ) and task_project.project_id in (select project_id from
project where stage='Appraise' and Business_stream = 'Upstream') and
task_project.page_id=page.page_id) as bmax, (select max(answer) from answer
where answer.page_id=page.page_id) as datamax, (select avg(avgscore) from
task_project where project_id=1 and task_project.page_id=page.page_id) as
projavg, (select avg(avgscore) from task_project where project_id not in
(select project_id from projectextra where subteam=1) and
task_project.page_id=page.page_id) as companyavg, (select avg(avgscore) from
task_project where project_id not in (select project_id from projectextra
where subteam=1) and project_id in (select project_id from project where
Business_stream = 'Upstream') and task_project.page_id=page.page_id) as
businessavg, page.* from page,riverorder where page.category_name='Business
Boundaries' and stage_name='Appraise' and
riverorder.category_name=page.category_name order by
riverorder.riverorder,page.order_id select (select max(answer.answer) from
answer where answer.member_id in ( select member_id from team_members where
project_id in ( select project_id from project where

26. It could be better.
yes? better is good!

27. {[-­‐]
checkin
:
{[-­‐]
badges
:
[],
created
:
1331454784,
geolat
:
"30.2640941786",
geolong
:
"-­‐97.7414819408",
mayor
:
{[-­‐]
type
:
"nochange"
},
primarycategory
:
{[-­‐]
fullpathname
:
"Food:American
Restaurants",
iconurl
:
"https://foursquare.com/img/categories/food/default.png",
id
:
"4bf58dd8d48988d14e941735", Text
nodename
:
"American
Restaurants"
},
timezone
:
"America/Chicago",
user
:
{[-­‐]
gender
:
"male"
},
venue
:
{[-­‐]
id
:
"4d752b1bba682d43e7563876",
name
:
"CNN
Grill
@
SXSW
(Max's
Wine
Dive)"
}
}
} readable, ya think?

28. source=foursquare | timechart count
by checkin.venue.name
The languages to talk to data are
getting better for us humans

29. Guys.. come on! Go
back to the data please.

31. Need data?
a simple way to describe a
massive problem
A friend in Boulder can help

32. The Social Media API
Jud Valeski
Co-Founder, CEO

34. Just when you think you’re all
done, wait. There is another
consumer you may have
forgotten

35. Someone with
a different
perspective
sees your
service as
input to theirs

36. DEMAND REALTIME DATA
IN A STREAM OVER THE WEB
IN JSON FORMAT

37. Hey audience!
We still have a few
minutes.
What questions
might you have
been saving until
this exact moment?

38. Thanks.
Erik Swan, CTO
Co-Founder,
Splunk
Michael Wilde
Splunk Ninja
Who else sends you on your way with a cute dog photo?