2. Who is Jon Rooney?
Director of Developer Marketing
Splunker since April 2012
Based in Splunk’s Seattle Office
Veteran of Microsoft, start-ups,
Accenture
2
4. Application Development Challenges
4
Build Deployment
Server
QA /
Test
Staging
Environment
Continuous
Integration /
Build Server
Source
Repository
Task Tracking
Local Build
5. Application Development Challenges
5
Build Deployment
Server
QA /
Test
Staging
Environment
Continuous
Integration /
Build Server
Source
Repository
Task Tracking
Local Build
Lack of visibility across the product
development lifecycle
Slows down the ability to detect and
troubleshoot errors
Limited visibility into application
usage and performance
6. 6
CI / Build
Servers
Project and Issue
Tracking
Code
Repository
QA / Testing
Tools
Splunk can solve these problems
Deployment
Servers
7. Splunk for Application Lifecycle Intelligence
7
Reduce Time to
Market
Resolve issues faster
Gain Agility
Improve Code Quality
and Lifecycle Visibility
Generate
Application Insights
Instrument
Analytics
11. Evolving the Splunk Platform
Collection
Indexing
Search Processing Language
Core Functions
Inputs, Apps, Other
Content
SDKs
Operational Intelligence Platform
Content
Core Engine
User and Developer Interfaces
Web Framework
REST API
12. Powerful Platform for Enterprise Developers
12
REST API
Build Splunk Apps Extend and Integrate Splunk
Simple XML
JavaScript
Django
Web
Framework
Java
JavaScript
Python
Ruby
C#
PHP
Data Models
Search Extensibility
Modular Inputs
SDKs
13. Splunk Web Framework
13
Familiar Technologies
- Code with JavaScript & Django/Python
- HTML5/CSS/JS Support
- Built on JQuery & Backbone.js
Flexible and Extensible
- Create custom layouts
- Integrate visualizations like Sankey
charts, heat maps and bubble charts
- SimpleXML to HTML Conversion
Simple XML
JavaScript
Django
Web
Framework
14. Log directly to
Splunk via TCP,
UDP, HTTP
Integrate search
results with other
applications using
custom
visualizations
Create and run
searches from
other applications
The REST API and SDKs
14
VisualizeSearch Manage
Add/Delete Users
Manage Inputs
Index
16. Who is Damien Dallimore
16
Worldwide Developer Evangelist @ Splunk
I develop
I talk about developing
Helping to build the Splunk developer ecosystem
Came from the Splunk Community
Once was a customer
Coder
17. I develop
17
All 100% free and open sourced , published to Splunk Apps , source code on Github
18. An Open Platform for Developers
18
Splunk is an open and extensible platform at numerous different
touchpoints for developers
Extensibility creates ecosystems
19. A Developer’s Smorgasboard
19
• CLI
• REST API
• 6 language SDKs
• Splunkbase Apps / Add-ons
• Custom search commands
• Scripted Inputs
• Modular Inputs
• The Web Framework
• Standard HTML/Javascript/CSS
• Data Models
• External scripted lookups
• Custom REST Endpoints
• Tools , utilitys and librarys
• Integrations with other software
frameworks (Spring)
• Hadoop dev with HUNK
• Custom user interfaces /
visualizations
• Mobile with Bugsense
• Custom data connectors (ODBC)
• Custom authentication handlers
20. Which Splunk Product for Devs ?
20
Splunk> Enterprise : Free to download and use. Index 500 MB/day.
Splunk> Cloud : Premium, cloud hosted. Full Enterprise stack.
Splunk> AMIs : BYOL versions for Amazon AWS Cloud.
Hunk> : Splunk for data in Hadoop clusters. Same platform, same easy apps.
Splunk> Storm : Free to use, cloud hosted. 20GB/30days.
22. Modular Inputs
22
• Extend the Splunk framework to define a custom input capability, just like the standard inputs you are
familiar with (TCP/UDP/File etc…)
• Splunk treats your custom input definitions as if they were part of Splunk's native inputs, totally
integrated first class citizen objects in Splunk
• Users interactively create and update your custom inputs using Splunk manager, just as they do for
native inputs.
• When deploying without a UI , you push out the inputs.conf file.
• All the properties are fully manageable via the REST API
• Version 5.0 +
24. Developing
24
• My preference is to use Python, however any language can be used.
• http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModInputsIntro
• There is a certain amount of “plumbing” to put in place , so I like utilities that take care of this for
you, so you can just focus on the business logic.
• Java,Python,C# SDKs also have Modular Input APIs
• Eclipse plugin has a wizard for creating Modular Inputs in Java
25. REST : The Data Potential
25
• Twitter
• Foursquare
• LinkedIn
• Facebook
• Fitbit
• Amazon
• Yahoo
• Reddit
• YouTube
• Flickr
• Wikipedia
• GNIP
• Box
• Okta
• Datasift
• Google APIs
• Weather Services
• Seismic monitoring
• Publicly available socio-economic data
• Traffic data
• Stock monitoring
• Security service providers
• Proprietary systems and platforms
• Other “data related” software products
• The REST “dataverse” is vast , but I think you get
the point.
There is a world of data out there available via REST that can be brought into Splunk, correlated and
enriched against your existing data, or used for entirely new uses cases that you might conceive of once you
see what is available and where your data might take you.
26. You are only limited by your own “data imagination”
26
29. Splunk Search
29
• Splunk’s search and querying language is called SPL
• Allows you to search, analyze and manipulate your data.
• Designed with the “unix pipeline” in mind – a “search pipeline”
• From a (simplistic) mental point of view:
– Series of commands
– Each commands takes the input of the previous one
– Each command outputs a sparse table
30. Splunk Search - Example 1:
30
“index=_internal foo | eval bar=... | stats count by bar”
• Initial command fetches result from index/raw data store, and outputs a
table, with a row for each event, and a column for each field (not all rows
have all columns)
• Second command adds a new column to each row, “bar”
• Third command looks at all the unique values of “bar”, and counts how
many rows each value has.
• Note that the last command completely transformed the table, hence it is a
“transforming” or “non-streaming” command.
• The second command was merely additive, known as a “streaming”
command.
• You can also have “generating” commands ie: inputlookup
31. Custom Search Commands
31
• Just like the “eval” or “stats” commands, you can write your own
“search commands”.
• Python scripts which take data on stdin, and output data on stdout
• Data comes in/out as CSV (with special handling for MV fields)
• Many of Splunk’s builtin commands are written as Python scripts
(e.g. head, return, transpose)
32. Custom Search Commands: Building Blocks
32
Custom search commands are built in two parts:
• A Python script containing the implementation
• An entry in commands.conf declaring configuring the command
• Splunk SDK for Python has librarys and examples for creating custom
search commands
35. Splunk Web Framework
35
Familiar Technologies
- Code with JavaScript & Django/Python
- HTML5/CSS/JS Support
- Built on JQuery & Backbone.js
Flexible and Extensible
- Create custom layouts
- Integrate visualizations like Sankey
charts, heat maps and bubble charts
- SimpleXML to HTML Conversion
Simple XML
JavaScript
Django
Web
Framework
36. Splunk JS Stack & Django Bindings Concepts
36
Managers
• Search Query Wrappers
• SearchManager and
SaveSearch Manager
• Include Search Parameters
• Available within Django or
JavaScript
37. Splunk JS Stack & Django Bindings Concepts
37
Splunk Views
• UI widgets
• Designed to work with Search
Managers
• Charts, Maps, Inputs, Timelin
e, TimePicker, etc.
38. Splunk JS Stack & Django Bindings Concepts
38
URL Maps & Django View
• URL Maps enable custom
routes
• Django Views, enable custom
logic to provide to templates
39. Splunk JS Stack & Django Bindings Concepts
39
Templates & Template Tags
• Templates enable quick
layout options
• Templates support
inheritance of other
templates
• Tags call a Python function
can be used for:
– Text manipulation
– Flow control
– Load external information
– … and more
40. Splunk JS Stack & Django Bindings Concepts
40
Data Binding using Tokens
• Token based data-binding
mechanism that keep shared
data in sync.
• Enables in-page interactivity
41. Web Framework Toolkit App
41
• Improved productivity for developing your own
Web Framework based apps
• Reusable Visualization
• Packaged to lower the effort of adding cool
visualizations
• Improved Developer focused Command Line
Tools
• Getting started templates
• Automates common tasks
• Includes examples pages demonstrating
advanced concepts.
42. SimpleXML converted to HTML
42
• Transition a Simple XML page to HTML/JS
• Option to overwrite/edit existing dashboard or create new
• Full Splunk JS Stack is available (Add custom viz, tokens, etc.)
• Does not include Django backend
• No support for visual dashboard editor or PDF printing
50. My Guiding Viz Principle
50
The visualization must be simple and intuitive to understand and derive meaning from at a glance.
Cool viz , but what are you telling me ?
51. So many options , which one for me ?
Splunk Web Form Editor
– If you are not a coder , not familiar with Simple XML
Edit Simple XML
– Familiar with Simple XML , what to customize more
Convert Simple XML to HTML/JS
– Coder , want to see the underlying JS/HTML , want custom UI behavior above Simple
XML, want to use some other JS/CSS
Simple XML JS Import
– Closely related to the above , perhaps you still want dashboard editing / PDF export
Django
– Previous benefits + want to leverage Django tags , want custom server side processing in
Django views
51
54. Company Overview
The right toolset for analyzing and troubleshooting mobile apps
in real-time
54
Deliver high quality, engaging apps.
Splunk BugSense
55. 55
Capabilities
Mobile Data
HTML5 dashboard
• Actionable reports
• Easy to use
How It Works
Cross Platform SDKs
• Install in < 5 min
• Crashes/Errors
• Events
• Sessions
• Transactions
Bug Sense Cloud
• Highly Scalable
• Secure
• Cloud Service
• Highly available
• Cost-effective
56. Integration (Android)
1. Add the .jar (download or use Maven/Gradle) & import
2. Add the permissions
3. Initialize:
61. WHY should you develop ?
61
Make money , Promote your company, Make sales !
Community and collaboration
Share / Give Back
Get a job / Build a career
Learn new skills / Educate yourself and others
Hadoop productivity
Do good
Open up new data sources for others to collaborate on
We talk a lot about the how , what , where and who ….. but what about the WHY
65. Where to go for More Info
• Tutorials, Code Samples, Getting Started, Downloads
– http://dev.splunk.com/
• Splunk Apps
– https://apps.splunk.com
• GitHub
– https://github.com/splunk/
• Twitter
– https://twitter.com/splunkdev
• Blogs
– http://blogs.splunk.com/dev/
65
66. The 5th Annual Splunk WWUC
• 50+ Customer speakers
• 30+ Apps in Splunk Apps
Showcase
• 30+ Technology Partners
• Ask The Experts
• Sales Meetings
• Business Value ROI booth
conf.splunk.com
• Las Vegas: Oct 6-9, 2014
• The MGM Grand Hotel
• 4000+ IT and Business Professionals
• 3 days of content, 130+ sessions
• 3 days of Splunk University
• Get Certified!
While Splunk is well-known as a world-class solution for IT and security professionals around the world, the Splunk platform
With the increased speed of product delivery – from an annual cadence with packaged software to continuous deployment with modern cloud services – organizations are under increasing pressure to run a well-oiled, fault tolerant, rapid delivery pipeline in their product development lifecycle. DevOps practices like continuous integration, automated configuration and continuous deployment increase the dependence of systems like task tracking and source code repositories with build servers and test suites.
With data moving rapidly through these different tools, it becomes challenging to maintain a grasp of the process. Despite challenges with silo-ed data in various tools, it is still critical to maintain full visibility of the product development journey – from user stories to production data. The various systems and tools contain the data necessary – like task data stored in JIRA, acceptance criteria and test results and the status of builds in QA and staging environments – is trapped in different formats in various tools, without an easily, flexible way to correlate and gain insights from.
Splunk increases the speed and efficiency of application development and testing, reducing time to market and enabling DevOps agility with connected visibility across the lifecycle. By starting at the beginning of the Product Development Lifecycle, you can track a problem in production all the way back to the code that was checked into the build and the developer responsible for the code. Using Splunk throughout the Product Development lifecycle allows the customers to be more agile and reduce time to market. Splunk enables continuous development and delivery of enterprise applications across the entire lifecycle, making the entire continuous delivery process seamless and frictionless for developers, testers, and operations personnel. Splunk deliver Application Lifecycle Intelligence – real time, mission critical visibility into every step, system and process involved in shipping new product to your customers.
Resolve Issues FasterSplunk enables developers to quickly trace errors to reduce time to market. As code moves from unit testing to staging environments, Splunk can be invaluable in tracking and benchmarking high-volume stress tests.Improve Code Quality and Lifecycle VisibilitySplunk provides visibility into events and activities in the disparate tools and systems that drive the development lifecycle, from task management and code repositories to build and deployment servers. Get real-time insights into automated processes like builds and tests to quickly identify errors to support rapid release cycles.Usage analyticsSplunk's late binding schema and powerful search language help developers analyze semantic data and deliver powerful business-level insights around usertrends, preferences, feature adoption, and more. By writing intelligence in the application logs through the practice of semantic logging, developers can use thelogs generated from custom applications to get powerful business insights without building or implementing special-purpose software. You have no controlover other systems events.With custom application development, you have full control over events that you write. While most events are written by developersto help them debug and some events are written to form an audit trail, semantic events are written explicitly for the gathering of analytics
We have many customers who have applied Splunk’s ability to collect, index and analyze disparate machine data in real time to the product development process, gaining critical Application Lifecycle intelligence.
As the Splunk platform evolves you can see how the Development capabilities have expanded to enable developers to better build on Splunk’s core capabilities. Let’s double click into the Developer platform….
BUILD SPLUNK APPSThe Splunk Web Framework makes building a Splunk app looks and feels like building any modern web application. The Simple Dashboard Editor makes it easy to BUILD interactive dashboards and user workflows as well as add custom styling, behavior and visualizations. Simple XML is ideal for fast, lightweight app customization and building. Simple XML development requires minimal coding knowledge and is well-suited for Splunk power users in IT to get fast visualization and analytics from their machine data. Simple XML also lets the developer “escape” to HTML with one click to do more powerful customization and integration with JavaScript. Developers looking for more advanced functionality and capabilities can build Splunk apps from the ground up using popular, standards-based web technologies: JavaScript and Django. The Splunk Web Framework lets developers quickly create Splunk apps by using prebuilt components, styles, templates, and reusable samples as well as supporting the development of custom logic, interactions, components, and UI. Developers can choose to program their Splunk app using Simple XML, JavaScript or Django (or any combination thereof).EXTEND AND INTEGRATE SPLUNKSplunk Enterprise is a robust, fully-integrated platform that enables developers to INTEGRATE data and functionality from Splunk software into applications across the organization using Software Development Kits (SDKs) for Java, JavaScript, C#, Python, PHP and Ruby. These SDKs make it easier to code to the open REST API that sits on top of the Splunk Engine. With almost 200 endpoints, the REST API lets developers do programmatically what any end user can do in the UI and more. The Splunk SDKs include documentation, code samples, resources and tools to make it faster and more efficient to program against the Splunk REST API using constructs and syntax familiar to developers experienced with Java, Python, JavaScript, PHP, Ruby and C#. Developers can easily manage HTTP access, authentication and namespaces in just a few lines of code. Developers can use the Splunk SDKs to: - Run real-time searches and retrieve Splunk data from line-of-business systems like Customer Service applications - Integrate data and visualizations (charts, tables) from Splunk into BI tools and reporting dashboards- Build mobile applications with real-time KPI dashboards and alerts powered by Splunk - Log directly to Splunk from remote devices and applications via TCP, UDP and HTTP- Build customer-facing dashboards in your applications powered by user-specific data in Splunk - Manage a Splunk instance, including adding and removing users as well as creating data inputs from an application outside of Splunk- Programmatically extract data from Splunk for long-term data warehousingDevelopers can EXTEND the power of Splunk software with programmatic control over search commands, data sources and data enrichment. Splunk Enterprise offers search extensibility through: - Custom Search Commands - developers can add a custom search script (in Python) to Splunk to create own search commands. To build a search that runs recursively, developers need to make calls directly to the REST API- Scripted Lookups: developers can programmatically script lookups via Python.- Scripted Alerts: can trigger a shell script or batch file (we provide guidance for Python and PERL).- Search Macros: make chunks of a search reuseable in multiple places, including saved and ad hoc searches. Splunk also provides developers with other mechanisms to extend the power of the platform.-Data Models: allow developers to abstract away the search language syntax, making Splunk queries (and thus, functionality) more manageable and portable/shareable. - Modular Inputs: allow developers to extend Splunk to programmatically manage custom data input functionality via REST.
Splunk is a fully-integrated platform that delivers rapid time-to-value to application developers. Developers can build robust applications on Splunk that deliver real-time business insights like clickstream analysis, IT early-warning systems, security and fraud protection at enterprise-grade scale using the languages, frameworks and tools that they know and love. Applications built on Splunk can deliver segmented, secure data to customers in any UI, powering your core product or service with real-time operational intelligence making Splunk more valuable and more relevant to more users. The Splunk Web Framework makes building a Splunk app looks and feels like building any modern web application. The Simple Dashboard Editor makes it easy to build interactive dashboards and user workflows as well as add custom styling, behavior and visualizations. Simple XML is ideal for fast, lightweight app customization and building. Simple XML development requires minimal coding knowledge and is well-suited for Splunk power users in IT to get fast visualization and analytics from their machine data. Simple XML also lets the developer “escape” to HTML with one click to do more powerful customization and integration with JavaScript. Developers looking for more advanced functionality and capabilities can build Splunk apps from the ground up using popular, standards-based web technologies: JavaScript and Django. The Splunk Web Framework lets developers quickly create Splunk apps by using prebuilt components, styles, templates, and reusable samples as well as supporting the development of custom logic, interactions, components, and UI. Developers can choose to program their Splunk app using Simple XML, JavaScript or Django (or any combination thereof).Use Simple XML for fast, lightweight dashboard building as well as add custom styling, behavior and visualizationsUse JavaScript for client-side development Use Python and the Django framework for server-side development
The Splunk SDKs empower developers to deliver greater operational agility throughout the enterprise by making it easy to integrate data from Splunk with other applications. Splunk provides a fully-documented and supported REST API with nearly 200 endpoints that let developers programmatically index, search and visualize data in Splunk from any application. Splunk’s SDKs, built on that API, make it easy for developers to integrate data from Splunk with other applications across the enterprise, from custom-built mobile reporting apps to off-the-shelf CRM solutions. Splunk offers SDKs for Python, Java, JavaScript, PHP, Ruby and C#. Developers can use the Splunk SDKs to: Run real-time searches and retrieve Splunk data from line-of-business systems like Customer Service applications Integrate data and visualizations (charts, tables) from Splunk into BI tools and reporting dashboardsBuild mobile applications with real-time KPI dashboards and alerts powered by Splunk Log directly to Splunk from remote devices and applications via TCP, UDP and HTTPBuild customer-facing dashboards in your applications powered by user-specific data in Splunk Manage a Splunk instance, including adding and removing users as well as creating data inputs from an application outside of SplunkProgrammatically extract data from Splunk for long-term data warehousingInteract with data stored in Hadoop using HUNK
Not talking about these , but grab me laterAsk audience for who has developed , who are coders
Journey , get data in , search , visualize
Half the battle is often just getting the data in.Data first , sexy next.
Tesla joke
Refer to Twitter demo / sentiment analysis4SQ , create map viz , show html then segway into customsearch commands with haversine
Sentiment search command on twitter screen
Show code from searchcommands_app
Splunk is a fully-integrated platform that delivers rapid time-to-value to application developers. Developers can build robust applications on Splunk that deliver real-time business insights like clickstream analysis, IT early-warning systems, security and fraud protection at enterprise-grade scale using the languages, frameworks and tools that they know and love. Applications built on Splunk can deliver segmented, secure data to customers in any UI, powering your core product or service with real-time operational intelligence making Splunk more valuable and more relevant to more users. The Splunk Web Framework makes building a Splunk app looks and feels like building any modern web application. The Simple Dashboard Editor makes it easy to build interactive dashboards and user workflows as well as add custom styling, behavior and visualizations. Simple XML is ideal for fast, lightweight app customization and building. Simple XML development requires minimal coding knowledge and is well-suited for Splunk power users in IT to get fast visualization and analytics from their machine data. Simple XML also lets the developer “escape” to HTML with one click to do more powerful customization and integration with JavaScript. Developers looking for more advanced functionality and capabilities can build Splunk apps from the ground up using popular, standards-based web technologies: JavaScript and Django. The Splunk Web Framework lets developers quickly create Splunk apps by using prebuilt components, styles, templates, and reusable samples as well as supporting the development of custom logic, interactions, components, and UI. Developers can choose to program their Splunk app using Simple XML, JavaScript or Django (or any combination thereof).Use Simple XML for fast, lightweight dashboard building as well as add custom styling, behavior and visualizationsUse JavaScript for client-side development Use Python and the Django framework for server-side development
The Web Framework Toolkit ships with support for the following visualization libraries:Sankey ChartGlobeBubble ChartForce DirectedCal HeatParallel CoordinatesParallel SetsSunburst (Doughnut++)
Closely relatedsupport for visual dashboard editor or PDF printing
The Web Framework Toolkit ships with support for the following visualization libraries:Sankey ChartGlobeBubble ChartForce DirectedCal HeatParallel CoordinatesParallel SetsSunburst (Doughnut++)
Build even more compelling appsUniversal drilldownIndividual refreshPan/zoomMulti time range pickersChart overlayAxis label rotationMore from control options (checkboxes ,multi input selections )
Make moneyMake your Splunk App a commercial offering. You can list it on Splunk Apps and get the benefit of that web presence and traffic and then redirect users to your own hosted site where your App is monetized. Generate leads , sell splunk licenses , differentiate from the competition.Promote your companyMany Splunk Apps are for the products of our technology partners. Take advantage of this marketing platform , generate more leads for your business via your Splunk creations , sell more license revenue or generate more service revenue if you are a Splunk reseller/partner.Community and CollaborationBecome part of the Splunk developer ecosystem , collaborate on Splunk Apps and Github , creating exciting new Apps that utilize the power of the Splunk platform to tackle innovative new data solutions.Share / Give BackSplunk Apps has loads of great content , most of it free. So why not give back also if you are using a lot of this content.Get a job / Build a careerI’ve seen a lot of resumes over the years. What makes a technical IT resume stand out from the crowd for me is often your involvement in community and collaborative projects.Learn new skills / Educate yourselfThe techniques involved in building Splunk Apps and Add-ons expose you to a rich landscape of software development and general IT skills. Multiple language development, Web development, APIs, Networking skills , OS and Hardware skills, Maths and Statistics skills etc… Splunk is a fantastic IT learning platform. And what better way to get some practical skills than by building something.Hadoop productivityWith the introduction of HUNK , we have significantly lowered the barrier to get productive results with data locked away in HDFS.So you can build a Splunk App to deliver insights on whatever data you have in HDFS that you are finding difficult to unleash.Do goodWalk in the footsteps of our Splunk for Good initiative and use data and your Splunk App to do some good in this world. There is so much publicly available data that could be used for this purpose i.e.: as 1 such idea you could create an App that uses Social Data and Sentiment Analysis to detect and combat cyberbullying.Open up new data sources for others to collaborate onData can sometimes be hard to get at. A Splunk Add-on can make it easier for others to get at that data so they can start being productive. They may even reuse your Add-on as part of a grander Splunk App.Solve problems , make new discoveriesThe true essence of innovation , creating and discovering new things that have not been done before. Use Splunk to make new discoveries in data and bundle this up in a Splunk App for the community to springboard off.Oh , and one more reason , it’s really fun and rewarding to create things !! Especially with the tremendous resource of data that we have at our disposal , you are only limited by your own imaginations.
.conf2013 war our 3rd annual conferenceHeld in Las Vegas at The Cosmopolitan Hotel in September.Goal here is to make our customers smarter, because smarter customers find new ways to use Splunk and tell their colleagues to use Splunk. Specific conference goals:Help customers answer: Where will your data take you?Empower customers with knowledgeFoster deep, supportive relationships within the Splunk communityGarner rich feedback and input to create a better SplunkReinforce Splunk CommunityEquip Customers and Partners with skills for successCreate channel for sharing best practices—expanding use casesLive, in-person venue for trainingFoundation for everything Splunk--future Users’ Conferences, regional user groups, fueling Splunkbase and Splunk Answers…Successful customers=happy customers=more Splunk sales!