SlideShare ist ein Scribd-Unternehmen logo
1 von 19
Downloaden Sie, um offline zu lesen
Splunking the JVM
Damien Dallimore
Developer Evangelist
Copyright©2013,SplunkInc.
What is this JVM thing ?
2
• Circa 1991, Dr. James Gosling at Sun started developing a technology
for next generation smart devices/appliances
• “Green” became “Oak” which became “Java”
• Java 1.0 first appeared in January 1996.
• The JVM is a virtual machine that runs programs that are compiled
into Java bytecode
• Available for many hardware and software platforms
• 17 years later , the JVM has evolved from a consumer device
technology,to a browser oriented technology with the explosion of
the web , to now becoming deeply rooted in the enterprisesoftware
landscape on the server side and in the cloud
Copyright©2013,SplunkInc.
17 years later
3
• Oracle took ownership of Java from Sun in
January 2010
• The Java Community Process(JCP) is the
forum where members develop
specifications for Java technology
• Java Specification Requests(JSR) get
submitted for new features, are reviewed and
then voted on by the JCP Executive
committee.
• Editions
• Embedded Java, Java ME , Java SE , Java EE
• Current Version is Java 7 (Dolphin)
• Java 8 scheduled for 2013
Application Servers Enterprise Service Buses Databases
NoSQL Distributed Big Data Web Servers
Directory Servers Search Engines Build Systems
Gaming Platforms Trading Systems Reservation Systems
Core Banking Messaging Infrastructure Proprietary Systems
Copyright©2013,SplunkInc.
JVM Variants
4
• Oracle Hotspot (formerly SUN)
– theprimaryreferenceJVMimplementation
• Oracle JRockit (formerly BEA)
– freesinceMay2011
– codebasecurrentlybeingmergedwithHotspot,ETA~JDK 8
• Open JDK
– SUN opensourcedHotspotand the Java classlibraryin 2006
– SlightdifferenceswithOracleJava still
– OpenJDKis the official JavaSE7 ReferenceImplementation
• J9
– IBM’sJVMforAIX,Linux,MVS, OS/400, PocketPC, z/OS
• Azul Systems Zing
– basedonHotSpot
– supportsmemoryheapsup to 512 GB withoutGCpausesand is ableto growand shrinkthe heap
basedonload
Copyright©2013,SplunkInc.
The JVM has a healthy future
5
• Hotspot/JRockitcodemergecreating abestofbreedJVM,OracletocontributethistoOpenJDK
• OpenJDKisthriving,OraclearecontributingandbeinggoodstewardsofJava(despiteinitialskepticism)
• Proliferationof alternativeJVMlanguagesthatcanallco-habitateintheJVMandnewfeaturesinJava8tofurtherenhance
thismultilanguageplatform
– Scala
– Groovy
– Clojure
• TheJVMisevolvingorganicallywiththeshiftingtidesofEnterprisesoftware,itisn’taboutthe“J”anymore.
• FromtheclusteredApplicationServerdominationofthe00’swenowseeanexplosionofBigDataproductsrunningin
massivelydistributedenvironmentsoncommodityhardwareorinthecloud
– ApacheHadoopfamily(MapReduce,Hive,Hbase,Cassandra,HDFS)
Copyright©2013,SplunkInc.
What is running in JVMs ?
6
Copyright©2013,SplunkInc.
JVM “Fanboi”
7
Dr. GoslingFanboi
Speaking of Java as a language as opposed
to the JVM platform, James Gosling, the
Father of Java, said "Most people talk
about Java the language, and this may
sound odd coming from me, but I could
hardly care less."
He went on to explain, "What I really care
about is the Java Virtual Machine as a
concept, because that is the thing that ties
it all together."
Copyright©2013,SplunkInc.
JVM Machine Data
8
• The JVM footprint cross cuts the data centre and represents a massive source of valuable machine data
• Large scale Application/Web Server clusters
• Hadoop & Cassandra Node topologies in the 10’000s !!!
Custom Developed
Code
WAR file
Application Code
Tomcat
JVM
Hotspot
Operating System
Linux
JMX, SNMP, HPROF,GC Logs, Custom Agents(AppDynamics/SplunkJavaAgent)
JMX, Application Logs
JMX, Developer Logs, Splunk Java SDK, SplunkJavaLogging
JVM process OS resource metrics
CORRELATE
Copyright©2013,SplunkInc.
Application & Developer Logs
9
• Application logs
• default logs that are part of the product
• Developer logs
• any custom code created and deployed
to the application that has it’s own
logging
• Written to local disk or a mounted network
volume
• Monitor with a Splunk UF
Splunk Indexer
Splunk Universal Forwarder
Monitor Log Files/ Directorys
Developed Code
Application
JVM
OS
Copyright©2013,SplunkInc.
Splunk Java SDK / SplunkJavaLogging
10
Splunk Indexer
Developed Code
Application
JVM
OS
HTTP$REST$/$TCP$/$UDP • Alternative to writing to log file or
needing to deploy a Splunk Universal
Forwarder
• Use the Splunk Java SDK to input events
directly to Splunk via HTTP Rest.
• Use SplunkJavaLogging to input events
directly to Splunk using custom logging
appenders.
Copyright©2013,SplunkInc.
JVM Process OS Metrics
11
• By JVM Process ID : Process State, Memory, CPU,
Disk Usage, Disk I/O, Network I/O, File
Descriptor Usage.
• Some OS metrics also exposed via JMX
• Splunk for Unix and Linux
• Splunk for Windows
• Correlate this OS data across your JVM and
Application events ie: your JVM may have hung
because of CPU starvation caused by some other
process thrashing
Splunk Indexer
Splunk for Unix or Linux
Monitor Log Files &
Directorys
Developed Code
Application
JVM
OS
Poll output from
commands
Copyright©2013,SplunkInc.
Garbage Collection logs
12
Splunk Indexer
Splunk Universal Forwarder
Monitor GC Log Files
Developed Code
Application
JVM
OS
• Extended Hotspot JVM options
-verbose:gc
-Xloggc:/home/damien/jvm_logs/gc.log
-XX:+PrintGC
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
• The log is written to at Garbage Collection time
• Useful for tracing full GC cycles
• Need to perform field extractions in Splunk
• Many GC metrics also available via JMX
54.736: [Full GC 54.737:
[Tenured: 172798K->18092K(174784K), 2.3792658 secs] 257598K->18092K(259584K),
[Perm : 20476K->20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs]
Copyright©2013,SplunkInc.
Custom Instrumentation Agents (Advanced)
13
Splunk Indexer
Splunk Universal
Forwarder
Monitor Agent Log Files
Developed Code
Application
JVM
OS
REST/TCP/UDP
• JVM BCI (byte code instrumentation)
• Write custom agents that get injected into
the running JVM
• Dynamically inspect the state of
applications running in the JVM
• Profiling, debugging, monitoring,
thread/memory analysis
• As you write the agent code , the data
output can be file based or over the
network
• Check out my SplunkJavaAgent on github
• Also AppDynamics have some pretty cool
kung fu in this area, we integrate !
Copyright©2013,SplunkInc.
HPROF Profiling Dumps
14
Splunk Indexer
Splunk Universal
Forwarder
Binary HPROF dump file
Developed Code
Application
JVM
OS
Monitor and decode into
textual key=value pairs
• Binary JVM dumps that allow for deeper JVM resource
inspection
• Typical use case is diagnosing memory issues after JVM
crashes with java.lang.OutOfMemoryError
• Binary file is usually batch loaded into a third party
memory analysis tool like Eclipse MAT
• SplunkJavaAgent can dynamically dump and decode
hprof output and send to Splunk
• Awesome source of information for dev/test
Warning : heap dumping is an expensive operation as a full GC gets performed
Copyright©2013,SplunkInc.
SNMP
15
• The JVM SNMP Agent provides a single MIB that exposes the
JVM’s Management and Monitoring API
http://docs.oracle.com/javase/1.5.0/docs/guide/management/JVM-MANAGEMENT-MIB.mib
• Setup the JVM (just the basic settings shown)
Open a UDP Port : -Dcom.sun.management.snmp.port=9004
Configure the ACL : $JAVA_HOME/jre/lib/management/snmp.acl
• Traps can be caught locally to file and monitored
• Splunk SNMP Modular Input can poll the JVM SNMP Objects
(coming soon to a theatre near you)
Splunk Indexer
Splunk Universal
Forwarder
Developed Code
Application
JVM
OS
SNMP%Objects%
Polled
JVM MIB
snmptrapd UDP:162
SNMP%Traps%
wri6en%to%file
Copyright©2013,SplunkInc.
JMX (Java Management Extensions)
16
Splunk Indexer
Developed Code
Application
JVM
OS
Splunk Universal
Forwarder
JMX
• Manage and Monitor the JVM and Application via
exposed MBeans
• JVM MBeans (java.lang domain)
• Vendor MBeans (most vendors ship their products with
extensive MBean coverage)
• Custom Coded MBeans (whatever your devs wish to
code)
• MBeans expose attributes, operations and
notifications to give you a powerfully dynamic
insight into the runtime state of the JVM and your
application.
• Add Splunk to the mix for historical and realtime
operational visibility, pro-active issue detection
etc..
• Splunk for JMX app on SplunkBase
Copyright©2013,SplunkInc.
JMX vs SNMP
17
JMX
• Open and easily extensible
• Developers can simply create new MBeans
• Vendor products(JBoss, Cassandra, Hadoop etc..) ship with thorough MBean coverage, not MIBs
SNMP
• The built-in SNMP agent of the JVM is not extensible.
• You will not be able to use it in order to expose your own custom MIB
• If you do want to expose your own MIB, you’d have to create a custom agent
Copyright©2013,SplunkInc.
Putting it all together, JVM Splunking Nirvana
18
Splunk Indexer Cluster
Developed Code
Application
JVM
OS
JMX
HPROF
OS*Metrics/Logs
Splunk Forwarder
Logs
JMX
REST/TCP/UDP
Auto Load Balanced
JMX
Logs
Distributed Search
Copyright©2013,SplunkInc.
Contact me
19
Email : ddallimore@splunk.com
Twitter : @damiendallimore
Skype : damien.dallimore
Github : damiendallimore
Splunkbase : damiend
Slideshare : http://www.slideshare.net/damiendallimore
Blogs : http://blogs.splunk.com/dev
Web : http://dev.splunk.com

Weitere ähnliche Inhalte

Was ist angesagt?

Seccomp Profiles and you: A practical guide.
Seccomp Profiles and you: A practical guide.Seccomp Profiles and you: A practical guide.
Seccomp Profiles and you: A practical guide.Duffie Cooley
 
Docker introduction
Docker introductionDocker introduction
Docker introductionPhuc Nguyen
 
IBM MQ - better application performance
IBM MQ - better application performanceIBM MQ - better application performance
IBM MQ - better application performanceMarkTaylorIBM
 
Devops & Configuration management tools
Devops & Configuration management toolsDevops & Configuration management tools
Devops & Configuration management toolsSonu Meena
 
Getting started with Docker
Getting started with DockerGetting started with Docker
Getting started with DockerRavindu Fernando
 
VXLAN Integration with CloudStack Advanced Zone
VXLAN Integration with CloudStack Advanced ZoneVXLAN Integration with CloudStack Advanced Zone
VXLAN Integration with CloudStack Advanced ZoneYoshikazu Nojima
 
Docker Birthday #3 - Intro to Docker Slides
Docker Birthday #3 - Intro to Docker SlidesDocker Birthday #3 - Intro to Docker Slides
Docker Birthday #3 - Intro to Docker SlidesDocker, Inc.
 
공개소프트웨어 기반 주요 클라우드 전환 사례
공개소프트웨어 기반 주요 클라우드 전환 사례공개소프트웨어 기반 주요 클라우드 전환 사례
공개소프트웨어 기반 주요 클라우드 전환 사례rockplace
 
AIXpert - AIX Security expert
AIXpert - AIX Security expertAIXpert - AIX Security expert
AIXpert - AIX Security expertdlfrench
 
Start your adventure with docker
Start your adventure with dockerStart your adventure with docker
Start your adventure with dockerSagar Dash
 
Docker and kubernetes_introduction
Docker and kubernetes_introductionDocker and kubernetes_introduction
Docker and kubernetes_introductionJason Hu
 
Introduction to Docker Compose
Introduction to Docker ComposeIntroduction to Docker Compose
Introduction to Docker ComposeAjeet Singh Raina
 
Introduction to docker
Introduction to dockerIntroduction to docker
Introduction to dockerJohn Willis
 
Troubleshooting Strategies for CloudStack Installations by Kirk Kosinski
Troubleshooting Strategies for CloudStack Installations by Kirk Kosinski Troubleshooting Strategies for CloudStack Installations by Kirk Kosinski
Troubleshooting Strategies for CloudStack Installations by Kirk Kosinski buildacloud
 
IBM MQ on cloud and containers
IBM MQ on cloud and containersIBM MQ on cloud and containers
IBM MQ on cloud and containersRobert Parker
 
Introduction to Docker
Introduction to DockerIntroduction to Docker
Introduction to DockerAditya Konarde
 

Was ist angesagt? (20)

Jboss Tutorial Basics
Jboss Tutorial BasicsJboss Tutorial Basics
Jboss Tutorial Basics
 
Seccomp Profiles and you: A practical guide.
Seccomp Profiles and you: A practical guide.Seccomp Profiles and you: A practical guide.
Seccomp Profiles and you: A practical guide.
 
Docker introduction
Docker introductionDocker introduction
Docker introduction
 
IBM MQ - better application performance
IBM MQ - better application performanceIBM MQ - better application performance
IBM MQ - better application performance
 
Devops & Configuration management tools
Devops & Configuration management toolsDevops & Configuration management tools
Devops & Configuration management tools
 
Getting started with Docker
Getting started with DockerGetting started with Docker
Getting started with Docker
 
VXLAN Integration with CloudStack Advanced Zone
VXLAN Integration with CloudStack Advanced ZoneVXLAN Integration with CloudStack Advanced Zone
VXLAN Integration with CloudStack Advanced Zone
 
Docker Birthday #3 - Intro to Docker Slides
Docker Birthday #3 - Intro to Docker SlidesDocker Birthday #3 - Intro to Docker Slides
Docker Birthday #3 - Intro to Docker Slides
 
공개소프트웨어 기반 주요 클라우드 전환 사례
공개소프트웨어 기반 주요 클라우드 전환 사례공개소프트웨어 기반 주요 클라우드 전환 사례
공개소프트웨어 기반 주요 클라우드 전환 사례
 
AIXpert - AIX Security expert
AIXpert - AIX Security expertAIXpert - AIX Security expert
AIXpert - AIX Security expert
 
Start your adventure with docker
Start your adventure with dockerStart your adventure with docker
Start your adventure with docker
 
Docker and kubernetes_introduction
Docker and kubernetes_introductionDocker and kubernetes_introduction
Docker and kubernetes_introduction
 
Introduction to Docker Compose
Introduction to Docker ComposeIntroduction to Docker Compose
Introduction to Docker Compose
 
Proxmox for DevOps
Proxmox for DevOpsProxmox for DevOps
Proxmox for DevOps
 
Introduction to docker
Introduction to dockerIntroduction to docker
Introduction to docker
 
Troubleshooting Strategies for CloudStack Installations by Kirk Kosinski
Troubleshooting Strategies for CloudStack Installations by Kirk Kosinski Troubleshooting Strategies for CloudStack Installations by Kirk Kosinski
Troubleshooting Strategies for CloudStack Installations by Kirk Kosinski
 
VMWARE ESX
VMWARE ESXVMWARE ESX
VMWARE ESX
 
IBM MQ on cloud and containers
IBM MQ on cloud and containersIBM MQ on cloud and containers
IBM MQ on cloud and containers
 
Hypervisors
HypervisorsHypervisors
Hypervisors
 
Introduction to Docker
Introduction to DockerIntroduction to Docker
Introduction to Docker
 

Ähnlich wie Splunking the JVM

Splunking the JVM (Java Virtual Machine)
Splunking the JVM (Java Virtual Machine)Splunking the JVM (Java Virtual Machine)
Splunking the JVM (Java Virtual Machine)Damien Dallimore
 
JVMs in Containers - Best Practices
JVMs in Containers - Best PracticesJVMs in Containers - Best Practices
JVMs in Containers - Best PracticesDavid Delabassee
 
[JOI] TOTVS Developers Joinville - Java #1
[JOI] TOTVS Developers Joinville - Java #1[JOI] TOTVS Developers Joinville - Java #1
[JOI] TOTVS Developers Joinville - Java #1Rubens Dos Santos Filho
 
Simple tweaks to get the most out of your jvm
Simple tweaks to get the most out of your jvmSimple tweaks to get the most out of your jvm
Simple tweaks to get the most out of your jvmJamie Coleman
 
Simple tweaks to get the most out of your JVM
Simple tweaks to get the most out of your JVMSimple tweaks to get the most out of your JVM
Simple tweaks to get the most out of your JVMJamie Coleman
 
MWLUG - Universal Java
MWLUG  -  Universal JavaMWLUG  -  Universal Java
MWLUG - Universal JavaPhilippe Riand
 
SemeruRuntimesUnderTheCover .pptx
SemeruRuntimesUnderTheCover .pptxSemeruRuntimesUnderTheCover .pptx
SemeruRuntimesUnderTheCover .pptxSumanMitra22
 
D. Andreadis, Red Hat: Concepts and technical overview of Quarkus
D. Andreadis, Red Hat: Concepts and technical overview of QuarkusD. Andreadis, Red Hat: Concepts and technical overview of Quarkus
D. Andreadis, Red Hat: Concepts and technical overview of QuarkusUni Systems S.M.S.A.
 
Cannibalising The Google App Engine
Cannibalising The  Google  App  EngineCannibalising The  Google  App  Engine
Cannibalising The Google App Enginecatherinewall
 
Concierge - Bringing OSGi (back) to Embedded Devices
Concierge - Bringing OSGi (back) to Embedded DevicesConcierge - Bringing OSGi (back) to Embedded Devices
Concierge - Bringing OSGi (back) to Embedded DevicesJan S. Rellermeyer
 
ContainerWorkloadwithSemeru.pdf
ContainerWorkloadwithSemeru.pdfContainerWorkloadwithSemeru.pdf
ContainerWorkloadwithSemeru.pdfSumanMitra22
 
Trends and future of java
Trends and future of javaTrends and future of java
Trends and future of javaCsaba Toth
 
Java in a world of containers
Java in a world of containersJava in a world of containers
Java in a world of containersDocker, Inc.
 
Java in a World of Containers - DockerCon 2018
Java in a World of Containers - DockerCon 2018Java in a World of Containers - DockerCon 2018
Java in a World of Containers - DockerCon 2018Arun Gupta
 
Supercharge your Test & Dev Process with Ravello, Jenkins and the Cloud (Jenk...
Supercharge your Test & Dev Process with Ravello, Jenkins and the Cloud (Jenk...Supercharge your Test & Dev Process with Ravello, Jenkins and the Cloud (Jenk...
Supercharge your Test & Dev Process with Ravello, Jenkins and the Cloud (Jenk...Gil Hoffer
 
JavaOne2015-What's in an Object?
JavaOne2015-What's in an Object?JavaOne2015-What's in an Object?
JavaOne2015-What's in an Object?Charlie Gracie
 
#JavaOne What's in an object?
#JavaOne What's in an object?#JavaOne What's in an object?
#JavaOne What's in an object?Charlie Gracie
 

Ähnlich wie Splunking the JVM (20)

Splunking the JVM (Java Virtual Machine)
Splunking the JVM (Java Virtual Machine)Splunking the JVM (Java Virtual Machine)
Splunking the JVM (Java Virtual Machine)
 
JVMs in Containers
JVMs in ContainersJVMs in Containers
JVMs in Containers
 
JVMs in Containers - Best Practices
JVMs in Containers - Best PracticesJVMs in Containers - Best Practices
JVMs in Containers - Best Practices
 
[JOI] TOTVS Developers Joinville - Java #1
[JOI] TOTVS Developers Joinville - Java #1[JOI] TOTVS Developers Joinville - Java #1
[JOI] TOTVS Developers Joinville - Java #1
 
Simple tweaks to get the most out of your jvm
Simple tweaks to get the most out of your jvmSimple tweaks to get the most out of your jvm
Simple tweaks to get the most out of your jvm
 
Simple tweaks to get the most out of your JVM
Simple tweaks to get the most out of your JVMSimple tweaks to get the most out of your JVM
Simple tweaks to get the most out of your JVM
 
MWLUG - Universal Java
MWLUG  -  Universal JavaMWLUG  -  Universal Java
MWLUG - Universal Java
 
Linked Process
Linked ProcessLinked Process
Linked Process
 
SemeruRuntimesUnderTheCover .pptx
SemeruRuntimesUnderTheCover .pptxSemeruRuntimesUnderTheCover .pptx
SemeruRuntimesUnderTheCover .pptx
 
D. Andreadis, Red Hat: Concepts and technical overview of Quarkus
D. Andreadis, Red Hat: Concepts and technical overview of QuarkusD. Andreadis, Red Hat: Concepts and technical overview of Quarkus
D. Andreadis, Red Hat: Concepts and technical overview of Quarkus
 
Cannibalising The Google App Engine
Cannibalising The  Google  App  EngineCannibalising The  Google  App  Engine
Cannibalising The Google App Engine
 
JavaFX Uni Parthenope
JavaFX Uni ParthenopeJavaFX Uni Parthenope
JavaFX Uni Parthenope
 
Concierge - Bringing OSGi (back) to Embedded Devices
Concierge - Bringing OSGi (back) to Embedded DevicesConcierge - Bringing OSGi (back) to Embedded Devices
Concierge - Bringing OSGi (back) to Embedded Devices
 
ContainerWorkloadwithSemeru.pdf
ContainerWorkloadwithSemeru.pdfContainerWorkloadwithSemeru.pdf
ContainerWorkloadwithSemeru.pdf
 
Trends and future of java
Trends and future of javaTrends and future of java
Trends and future of java
 
Java in a world of containers
Java in a world of containersJava in a world of containers
Java in a world of containers
 
Java in a World of Containers - DockerCon 2018
Java in a World of Containers - DockerCon 2018Java in a World of Containers - DockerCon 2018
Java in a World of Containers - DockerCon 2018
 
Supercharge your Test & Dev Process with Ravello, Jenkins and the Cloud (Jenk...
Supercharge your Test & Dev Process with Ravello, Jenkins and the Cloud (Jenk...Supercharge your Test & Dev Process with Ravello, Jenkins and the Cloud (Jenk...
Supercharge your Test & Dev Process with Ravello, Jenkins and the Cloud (Jenk...
 
JavaOne2015-What's in an Object?
JavaOne2015-What's in an Object?JavaOne2015-What's in an Object?
JavaOne2015-What's in an Object?
 
#JavaOne What's in an object?
#JavaOne What's in an object?#JavaOne What's in an object?
#JavaOne What's in an object?
 

Mehr von Damien Dallimore

QCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT RodeoQCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT RodeoDamien Dallimore
 
Splunk Conf 2014 - Splunking the Java Virtual Machine
Splunk Conf 2014 - Splunking the Java Virtual MachineSplunk Conf 2014 - Splunking the Java Virtual Machine
Splunk Conf 2014 - Splunking the Java Virtual MachineDamien Dallimore
 
Splunk Conf 2014 - Getting the message
Splunk Conf 2014 - Getting the messageSplunk Conf 2014 - Getting the message
Splunk Conf 2014 - Getting the messageDamien Dallimore
 
SpringOne2GX 2014 Splunk Presentation
SpringOne2GX 2014 Splunk PresentationSpringOne2GX 2014 Splunk Presentation
SpringOne2GX 2014 Splunk PresentationDamien Dallimore
 
SplunkLive London 2014 Developer Presentation
SplunkLive London 2014  Developer PresentationSplunkLive London 2014  Developer Presentation
SplunkLive London 2014 Developer PresentationDamien Dallimore
 
Integrating Splunk into your Spring Applications
Integrating Splunk into your Spring ApplicationsIntegrating Splunk into your Spring Applications
Integrating Splunk into your Spring ApplicationsDamien Dallimore
 
Splunk Modular Inputs / JMS Messaging Module Input
Splunk Modular Inputs / JMS Messaging Module InputSplunk Modular Inputs / JMS Messaging Module Input
Splunk Modular Inputs / JMS Messaging Module InputDamien Dallimore
 
Splunk as a_big_data_platform_for_developers_spring_one2gx
Splunk as a_big_data_platform_for_developers_spring_one2gxSplunk as a_big_data_platform_for_developers_spring_one2gx
Splunk as a_big_data_platform_for_developers_spring_one2gxDamien Dallimore
 

Mehr von Damien Dallimore (14)

QCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT RodeoQCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT Rodeo
 
Splunk Conf 2014 - Splunking the Java Virtual Machine
Splunk Conf 2014 - Splunking the Java Virtual MachineSplunk Conf 2014 - Splunking the Java Virtual Machine
Splunk Conf 2014 - Splunking the Java Virtual Machine
 
Splunk Conf 2014 - Getting the message
Splunk Conf 2014 - Getting the messageSplunk Conf 2014 - Getting the message
Splunk Conf 2014 - Getting the message
 
SpringOne2GX 2014 Splunk Presentation
SpringOne2GX 2014 Splunk PresentationSpringOne2GX 2014 Splunk Presentation
SpringOne2GX 2014 Splunk Presentation
 
SplunkLive London 2014 Developer Presentation
SplunkLive London 2014  Developer PresentationSplunkLive London 2014  Developer Presentation
SplunkLive London 2014 Developer Presentation
 
A Brief History Of Data
A Brief History Of DataA Brief History Of Data
A Brief History Of Data
 
Integrating Splunk into your Spring Applications
Integrating Splunk into your Spring ApplicationsIntegrating Splunk into your Spring Applications
Integrating Splunk into your Spring Applications
 
Spring Integration Splunk
Spring Integration SplunkSpring Integration Splunk
Spring Integration Splunk
 
Splunk Modular Inputs / JMS Messaging Module Input
Splunk Modular Inputs / JMS Messaging Module InputSplunk Modular Inputs / JMS Messaging Module Input
Splunk Modular Inputs / JMS Messaging Module Input
 
Splunk for JMX
Splunk for JMXSplunk for JMX
Splunk for JMX
 
Splunk Java Agent
Splunk Java AgentSplunk Java Agent
Splunk Java Agent
 
Splunk Developer Platform
Splunk Developer PlatformSplunk Developer Platform
Splunk Developer Platform
 
Splunk as a_big_data_platform_for_developers_spring_one2gx
Splunk as a_big_data_platform_for_developers_spring_one2gxSplunk as a_big_data_platform_for_developers_spring_one2gx
Splunk as a_big_data_platform_for_developers_spring_one2gx
 
Using the Splunk Java SDK
Using the Splunk Java SDKUsing the Splunk Java SDK
Using the Splunk Java SDK
 

Kürzlich hochgeladen

20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 

Kürzlich hochgeladen (20)

20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 

Splunking the JVM

  • 1. Splunking the JVM Damien Dallimore Developer Evangelist
  • 2. Copyright©2013,SplunkInc. What is this JVM thing ? 2 • Circa 1991, Dr. James Gosling at Sun started developing a technology for next generation smart devices/appliances • “Green” became “Oak” which became “Java” • Java 1.0 first appeared in January 1996. • The JVM is a virtual machine that runs programs that are compiled into Java bytecode • Available for many hardware and software platforms • 17 years later , the JVM has evolved from a consumer device technology,to a browser oriented technology with the explosion of the web , to now becoming deeply rooted in the enterprisesoftware landscape on the server side and in the cloud
  • 3. Copyright©2013,SplunkInc. 17 years later 3 • Oracle took ownership of Java from Sun in January 2010 • The Java Community Process(JCP) is the forum where members develop specifications for Java technology • Java Specification Requests(JSR) get submitted for new features, are reviewed and then voted on by the JCP Executive committee. • Editions • Embedded Java, Java ME , Java SE , Java EE • Current Version is Java 7 (Dolphin) • Java 8 scheduled for 2013 Application Servers Enterprise Service Buses Databases NoSQL Distributed Big Data Web Servers Directory Servers Search Engines Build Systems Gaming Platforms Trading Systems Reservation Systems Core Banking Messaging Infrastructure Proprietary Systems
  • 4. Copyright©2013,SplunkInc. JVM Variants 4 • Oracle Hotspot (formerly SUN) – theprimaryreferenceJVMimplementation • Oracle JRockit (formerly BEA) – freesinceMay2011 – codebasecurrentlybeingmergedwithHotspot,ETA~JDK 8 • Open JDK – SUN opensourcedHotspotand the Java classlibraryin 2006 – SlightdifferenceswithOracleJava still – OpenJDKis the official JavaSE7 ReferenceImplementation • J9 – IBM’sJVMforAIX,Linux,MVS, OS/400, PocketPC, z/OS • Azul Systems Zing – basedonHotSpot – supportsmemoryheapsup to 512 GB withoutGCpausesand is ableto growand shrinkthe heap basedonload
  • 5. Copyright©2013,SplunkInc. The JVM has a healthy future 5 • Hotspot/JRockitcodemergecreating abestofbreedJVM,OracletocontributethistoOpenJDK • OpenJDKisthriving,OraclearecontributingandbeinggoodstewardsofJava(despiteinitialskepticism) • Proliferationof alternativeJVMlanguagesthatcanallco-habitateintheJVMandnewfeaturesinJava8tofurtherenhance thismultilanguageplatform – Scala – Groovy – Clojure • TheJVMisevolvingorganicallywiththeshiftingtidesofEnterprisesoftware,itisn’taboutthe“J”anymore. • FromtheclusteredApplicationServerdominationofthe00’swenowseeanexplosionofBigDataproductsrunningin massivelydistributedenvironmentsoncommodityhardwareorinthecloud – ApacheHadoopfamily(MapReduce,Hive,Hbase,Cassandra,HDFS)
  • 7. Copyright©2013,SplunkInc. JVM “Fanboi” 7 Dr. GoslingFanboi Speaking of Java as a language as opposed to the JVM platform, James Gosling, the Father of Java, said "Most people talk about Java the language, and this may sound odd coming from me, but I could hardly care less." He went on to explain, "What I really care about is the Java Virtual Machine as a concept, because that is the thing that ties it all together."
  • 8. Copyright©2013,SplunkInc. JVM Machine Data 8 • The JVM footprint cross cuts the data centre and represents a massive source of valuable machine data • Large scale Application/Web Server clusters • Hadoop & Cassandra Node topologies in the 10’000s !!! Custom Developed Code WAR file Application Code Tomcat JVM Hotspot Operating System Linux JMX, SNMP, HPROF,GC Logs, Custom Agents(AppDynamics/SplunkJavaAgent) JMX, Application Logs JMX, Developer Logs, Splunk Java SDK, SplunkJavaLogging JVM process OS resource metrics CORRELATE
  • 9. Copyright©2013,SplunkInc. Application & Developer Logs 9 • Application logs • default logs that are part of the product • Developer logs • any custom code created and deployed to the application that has it’s own logging • Written to local disk or a mounted network volume • Monitor with a Splunk UF Splunk Indexer Splunk Universal Forwarder Monitor Log Files/ Directorys Developed Code Application JVM OS
  • 10. Copyright©2013,SplunkInc. Splunk Java SDK / SplunkJavaLogging 10 Splunk Indexer Developed Code Application JVM OS HTTP$REST$/$TCP$/$UDP • Alternative to writing to log file or needing to deploy a Splunk Universal Forwarder • Use the Splunk Java SDK to input events directly to Splunk via HTTP Rest. • Use SplunkJavaLogging to input events directly to Splunk using custom logging appenders.
  • 11. Copyright©2013,SplunkInc. JVM Process OS Metrics 11 • By JVM Process ID : Process State, Memory, CPU, Disk Usage, Disk I/O, Network I/O, File Descriptor Usage. • Some OS metrics also exposed via JMX • Splunk for Unix and Linux • Splunk for Windows • Correlate this OS data across your JVM and Application events ie: your JVM may have hung because of CPU starvation caused by some other process thrashing Splunk Indexer Splunk for Unix or Linux Monitor Log Files & Directorys Developed Code Application JVM OS Poll output from commands
  • 12. Copyright©2013,SplunkInc. Garbage Collection logs 12 Splunk Indexer Splunk Universal Forwarder Monitor GC Log Files Developed Code Application JVM OS • Extended Hotspot JVM options -verbose:gc -Xloggc:/home/damien/jvm_logs/gc.log -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails • The log is written to at Garbage Collection time • Useful for tracing full GC cycles • Need to perform field extractions in Splunk • Many GC metrics also available via JMX 54.736: [Full GC 54.737: [Tenured: 172798K->18092K(174784K), 2.3792658 secs] 257598K->18092K(259584K), [Perm : 20476K->20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs]
  • 13. Copyright©2013,SplunkInc. Custom Instrumentation Agents (Advanced) 13 Splunk Indexer Splunk Universal Forwarder Monitor Agent Log Files Developed Code Application JVM OS REST/TCP/UDP • JVM BCI (byte code instrumentation) • Write custom agents that get injected into the running JVM • Dynamically inspect the state of applications running in the JVM • Profiling, debugging, monitoring, thread/memory analysis • As you write the agent code , the data output can be file based or over the network • Check out my SplunkJavaAgent on github • Also AppDynamics have some pretty cool kung fu in this area, we integrate !
  • 14. Copyright©2013,SplunkInc. HPROF Profiling Dumps 14 Splunk Indexer Splunk Universal Forwarder Binary HPROF dump file Developed Code Application JVM OS Monitor and decode into textual key=value pairs • Binary JVM dumps that allow for deeper JVM resource inspection • Typical use case is diagnosing memory issues after JVM crashes with java.lang.OutOfMemoryError • Binary file is usually batch loaded into a third party memory analysis tool like Eclipse MAT • SplunkJavaAgent can dynamically dump and decode hprof output and send to Splunk • Awesome source of information for dev/test Warning : heap dumping is an expensive operation as a full GC gets performed
  • 15. Copyright©2013,SplunkInc. SNMP 15 • The JVM SNMP Agent provides a single MIB that exposes the JVM’s Management and Monitoring API http://docs.oracle.com/javase/1.5.0/docs/guide/management/JVM-MANAGEMENT-MIB.mib • Setup the JVM (just the basic settings shown) Open a UDP Port : -Dcom.sun.management.snmp.port=9004 Configure the ACL : $JAVA_HOME/jre/lib/management/snmp.acl • Traps can be caught locally to file and monitored • Splunk SNMP Modular Input can poll the JVM SNMP Objects (coming soon to a theatre near you) Splunk Indexer Splunk Universal Forwarder Developed Code Application JVM OS SNMP%Objects% Polled JVM MIB snmptrapd UDP:162 SNMP%Traps% wri6en%to%file
  • 16. Copyright©2013,SplunkInc. JMX (Java Management Extensions) 16 Splunk Indexer Developed Code Application JVM OS Splunk Universal Forwarder JMX • Manage and Monitor the JVM and Application via exposed MBeans • JVM MBeans (java.lang domain) • Vendor MBeans (most vendors ship their products with extensive MBean coverage) • Custom Coded MBeans (whatever your devs wish to code) • MBeans expose attributes, operations and notifications to give you a powerfully dynamic insight into the runtime state of the JVM and your application. • Add Splunk to the mix for historical and realtime operational visibility, pro-active issue detection etc.. • Splunk for JMX app on SplunkBase
  • 17. Copyright©2013,SplunkInc. JMX vs SNMP 17 JMX • Open and easily extensible • Developers can simply create new MBeans • Vendor products(JBoss, Cassandra, Hadoop etc..) ship with thorough MBean coverage, not MIBs SNMP • The built-in SNMP agent of the JVM is not extensible. • You will not be able to use it in order to expose your own custom MIB • If you do want to expose your own MIB, you’d have to create a custom agent
  • 18. Copyright©2013,SplunkInc. Putting it all together, JVM Splunking Nirvana 18 Splunk Indexer Cluster Developed Code Application JVM OS JMX HPROF OS*Metrics/Logs Splunk Forwarder Logs JMX REST/TCP/UDP Auto Load Balanced JMX Logs Distributed Search
  • 19. Copyright©2013,SplunkInc. Contact me 19 Email : ddallimore@splunk.com Twitter : @damiendallimore Skype : damien.dallimore Github : damiendallimore Splunkbase : damiend Slideshare : http://www.slideshare.net/damiendallimore Blogs : http://blogs.splunk.com/dev Web : http://dev.splunk.com

Hinweis der Redaktion

  1. Open JDK – Browser plugin and web start only with Oracle Java