Graylog2 is a free and open source log analysis tool that allows you to perform searches, analyse the data, build dashboards and set alarms using the streams system. Typical use cases range from debugging platform problems & monitoring exception counts to displaying average pizza delivery time per state on a dashboard. In this talk I will go through the architecture of Graylog2, what you can do with it and how to get your data into it.

1. Graylog2
Lennart Koopmann, OSDC 2014
@_lennart / www.graylog2.org

2. About me
• 25 years old
• Living in Hamburg, Germany
• @_lennart on Twitter
• Co-Founder of TORCH - The Graylog2 company.

3. Graylog2 history
• Started as open source project by Lennart Koopmann in 2010 and
was developed entirely in free time.
• TORCH founded as company behind it in late 2012 after seeing
massive growth and worldwide distribution in large scale setups.
• Now team of 6 working full-time on it, three more people joining this
summer. (and still hiring)
• www.graylog2.org

4. Graylog2 history
• Big rewrite of Graylog2 started in 2012 and finished with releasing a
final v0.20.0 in February 2014 that addresses what we learnt from our
first customers and all users.
• Web Interface now (like the server) written in Java and easy to
install. Prior versions used Ruby On Rails and were hard to
install.
• New web Interface focussing on powerful analytics.
• Unified REST API communication for easy extending and
integrating with other products, tools and scripts.

5. Free and open source analysis of any
machine data written in your datacenter.
!
Running on the JVM in your own
environment. Not limited by licenses.

6. Basic architecture
message sources
graylog2-server graylog2-server
ElasticSearch
Cluster
MongoDB
graylog2-web-interface
Your own
reporting
scripts
Your own
subscribers
RESTREST
Inputs

7. Architecture considerations
• Use graylog2-radio for HA and high level buffering
• Put load balancers in front and scale out horizontally

8. Architecture considerations
• graylog2-server / graylog2-radio: Focus on CPU
• ElasticSearch: Focus on RAM and IO
• MongoDB: Replication set for failover, not much load
• graylog2-web-interface: Not much load at all

9. Architecture considerations
• http://support.torch.sh/help/kb/general/graylog2-architecture-high-level-overview

10. No message left behind
2014-04-04 14:05:43,147 INFO : org.graylog2.Core - SIGNAL received. Shutting down.
2014-04-04 14:05:43,150 INFO : org.graylog2.system.shutdown.GracefulShutdown - Graceful shutdown initiated.
2014-04-04 14:05:43,150 INFO : org.graylog2.system.shutdown.GracefulShutdown - Node status: [Halting [LB:DEAD]].
Waiting <5sec> for possible load balancers to recognize state change.
!
…
!
2014-04-04 14:05:49,156 INFO : org.graylog2.system.shutdown.GracefulShutdown - Attempting to close input
<org.graylog2.inputs.raw.udp.RawUDPInput.531f89283004f7b66a87e163> [Raw/Plaintext UDP].
2014-04-04 14:05:49,157 INFO : org.graylog2.system.shutdown.GracefulShutdown - Input
[org.graylog2.inputs.raw.udp.RawUDPInput.531f89283004f7b66a87e163] closed. Took [1ms]
…
!
2014-04-04 14:05:49,158 INFO : org.graylog2.caches.Caches - Waiting until all caches are empty.
2014-04-04 14:05:49,158 INFO : org.graylog2.caches.Caches - All caches are empty. Continuing.
2014-04-04 14:05:49,159 INFO : org.graylog2.buffers.Buffers - Waiting until all buffers are empty.
2014-04-04 14:05:49,159 INFO : org.graylog2.buffers.Buffers - All buffers are empty. Continuing.
!
…
!
2014-04-04 14:05:49,176 INFO : org.graylog2.system.shutdown.GracefulShutdown - Goodbye.

11. No message left behind

12. GELF http://graylog2.org/gelf
• The “Graylog2 Extended Log Format”. Structured and compressed,
based on JSON.
• Optional UDP chunking allows sending a lot of data without having
to care about connection management in your application (timeouts,
…) if you don’t need transport security.
• Already over 30 libraries from the community and integrated into the
first products.

13. Streams
• Performant realtime routing of messages based on rules. Matching applied
when the message is received and processed.
• Create streams like “SSH logins” or “Exceptions in application X” for quick
access in the web interface (like saved searches) or alerts.
• Be alerted based on message count thresholds or results of statistical
computation of given relative time windows. “Send me an alert when the
standard deviation of the response time in application X was higher than 100
in the last 10 minutes.”
• Forward to other systems based on matched streams. “Forward all business
intelligence related logs to another system. (to save license costs)”