How to think the zabbix way

1. How to think the zabbix wayHow to think the zabbix way
By @supergrilo
@tiagodacruz

2. Fábio SantosFábio Santos
A.K.A. Supergrilo

3. Tiago CruzTiago Cruz

4. Apps that make your life easier
For You
For Companies

7. BriefBrief
1. Timeout
2. Active vs Passive
3. Type of items
4. Trapper
5. Server and Proxy internals
6. Action
7. LLD

8. The beginning will decideThe beginning will decide
everything!everything!

9. Questions...Questions...
1. How large is my
environment?
2. How large my environment
can be?
3. How fast my environment
can grow

10. TimeoutTimeout
Specifies how long we wait
for agent, SNMP device or
external check (in seconds).
Spend no more than
Timeout seconds on
processing

11. How large?How large?

13. Active vs PassiveActive vs Passive

14. PassivePassive

15. ActiveActive

16. Item typesItem types
which is best?

17. More OverheadMore Overhead
Zabbix Agent
data gathering
SNMP
All protocol versions
IPMI
Intelligent Platform Management Interface
SIMPLE Checks
Network services
Internals checks
To monitoring internals of Zabbix
(SSH|Telnet) checks
agent-less monitoring
External Checks
check executed by Zabbix server
Aggregate checks
aggregate information
Calculated Items
calculations on the basis of other items
JMX
monitor JMX counters of a Java application

18. Less OverheadLess Overhead
Zabbix Agent (active)
data gathering using agent resources
SNMP Trapper
Receiving SNMP traps
Log Monitoring
monitoring and analysis of log files
Trapper
items accept incoming data
ODBC
Database monitor

19. TrapperTrapper

20. Don't do itDon't do it
$ cat /etc/zabbix/zabbix_agentd.d/too_slow.conf
UserParameter=check.tooslow,sleep 30

21. Zabbix SenderZabbix Sender
Running handlers:
[2016-04-16T12:56:33+00:00] INFO: Running report handlers
[2016-04-16T12:56:33+00:00] INFO: Zabbix::Report handler started
[2016-04-16T12:56:33+00:00] DEBUG: Sending tozabbix: angostura1.datac.movile.com custo
m.chef-client.last_run.success 1
angostura1.datac.movile.com custom.chef-client.last_run.elapsed_time 27.773516757
angostura1.datac.movile.com custom.chef-client.last_run.start_time 1460811365
angostura1.datac.movile.com custom.chef-client.last_run.end_time 1460811393
angostura1.datac.movile.com custom.chef-client.last_run.all_resources_num 450
angostura1.datac.movile.com custom.chef-client.last_run.updated_resources_num 16
[2016-04-16T12:56:33+00:00] DEBUG: Command /usr/bin/zabbix_sender --config /etc/zabbix
/zabbix_agentd.conf --input-file /tmp/client-handler-zabbix-report20160416-18836-w84yu
q
[2016-04-16T12:56:33+00:00] DEBUG: output ["info from server: "processed: 6; failed:
0; total: 6; seconds spent: 0.000168"n", "sent: 6; skipped: 0; total: 6n"]
- Zabbix::Report
Running handlers complete
[2016-04-16T12:56:33+00:00] INFO: Report handlers complete

22. InternalsInternals

23. Server InternalsServer Internals

24. Proxy InternalsProxy Internals

25. Server GatheringServer Gathering

26. Server CacheServer Cache

27. ActionsActions
Auto Registration
Trigger

28. * Event Source: Auto Registration
We use Chef o/ to generate the
zabbix_agentd.conf
<% if node.chef_environment == "production" %>
HostMetadata=Linux <%= node['base']['dc'] %> b41c15b8-c43f-42ba-8158
<% else %>
HostMetadata=Linux <%= node.chef_environment %>
<% end %>
And we use actions to register the machine in the correct
environment based on your datacenter

29. * Event Source: Triggers
- Slack
https://github.com/tiago-cruz-movile/zabbix-slack-
alertscript

30. * Event Source: Triggers
- Slack

31. * Event Source: Triggers
- OpsGenie
"We provide the tools you need to design meaningful,
actionable alerts and ensure the right people are notified."

32. Low Level DiscoveryLow Level Discovery

33. - Discovery rules
Name: ELBs in us-east-1
Type: External Check
Key : check_elb["list_elb","us-east-
1"]
Update Interval: 600
Keep lost resources period : 0
Template AWS ELBTemplate AWS ELB

34. - Item prototypes
Name: ELB Status {#ELB_NAME} - us-east-1
Type: External Check
Key : check_elb["elb_status_verify",us-east-1,{#ELB_NAME}]
Update Interval: 300
Type of Information: Numeric (float)
Units: %
[root@zabbix externalscripts]# ./check_elb elb_status_verify us-east-1 payment-api
100.00
[root@zabbix externalscripts]# ./check_elb elb_status_verify us-east-1 notifications
66.00

35. Discovered Items
- Trigger

36. JMX DiscoveryJMX Discovery
https://github.com/supergrilo/pyjmx-discovery

37. CaveatsCaveats

38. IssuesIssues
Zabbix links should be always unique to share with other people (Priority: Trivial)
https://support.zabbix.com/browse/ZBXNEXT-3179 (Mar/2016)
OK: /nagios/cgi-bin/extinfo.cgi?type=2&host=HOSTNAME&service=SERVICE
OK: /zabbix/charts.php?sid=aac18a95ab092ac1&form_refresh=1&fullscreen=0&groupid=5&hostid=13222&graphid=35034
NOK: /zabbix/charts.php?ddreset=1&sid=aac18a95ab092ac1
Search/ Filter on templates page (Priority: Trivial)
https://support.zabbix.com/browse/ZBXNEXT-3175 (Mar/2016)
Filter for Screens (Priority: Trivial)
https://support.zabbix.com/browse/ZBXNEXT-2189 (Mar/2014)
Multiline regular expression matching (Priority: Major)
https://support.zabbix.com/browse/ZBXNEXT-968 (Dec/2007)

39. - Proxy Last See
- No search on templates

40. Questions?Questions?
fabio.santos@movile.com
tiago.cruz@movile.com