1) The document discusses how monitoring micro-metrics like garbage collection logs and thread dumps can help predict production outages in applications. It provides examples of how specific micro-metrics could predict issues like memory leaks, backend slowdowns, CPU spikes, and poor response times. 2) The document also describes yCrash, a tool that captures micro-metrics every 3 minutes from applications and uses machine learning to detect potential problems and trigger full troubleshooting if an issue is forecasted. It provides open-source scripts to collect various system and application metrics for troubleshooting. 3) Real-world case studies are presented on how micro-metrics helped predict and solve issues for major financial, trading, and travel companies to prevent production

1. Predicting Production Outages:
Unleashing the Power of Micro-Metrics
Ram Lakshmanan
Architect yCrash

2. Predicting Memory Problems

3. 3
Memory of Healthy Application
- Full Garbage Collection Event

4. 4
Acute Memory Leak
- Full Garbage Collection Event

5. 5
Memory Leak
- Full Garbage Collection Event

6. GC Throughput
Micrometric
Source: Garbage Collection Log

7. 7
What is GC Throughput?
Amount of time application spends in processing customer
transactions
vs
Amount of time application spends in processing garbage
collection activity

8. 1. GC Log
10. netstat
12. vmstat
2. Thread Dump
9. dmesg
3. Heap Dump
6. ps
8. Disk Usage
5. top 13. iostat
11. ping
14. Kernel Params
15. App Logs
16. metadata
4. Heap Substitute
7. top -H
8
Open-source script:
https://github.com/ycrash/yc-data-script
360° Troubleshooting artifacts

9. Predicting Backend Slowdown

10. Application Architecture
JDBC
SOAP
MainFrame
REST
Server Thread Pool
Application Server
HTTP(S) request
10

11. Application Architecture
JDBC
SOAP
MainFrame
REST
Server Thread Pool
Application Server
HTTP(S) request
11

12. Threads with identical Stack trace
Micrometric
Source: Thread Dump

13. 13
Case Study
Backend Slowdown in a Major
Financial Institution in N.
America

14. Predicting CPU Spike

15. What Causes CPU to Spike?
15

16. top –H –p <PROCESS_ID>’
Secrete Option:
16
We all might have used ‘top’

17. Thread Level CPU consumption
Micrometric
Source: top –H –p <PROCESS_ID>

18. Case Study
Major Trading app in N.
America
https://blog.fastthread.io/2020/04/23/troubleshooting-cpu-spike-in-a-major-trading-application/
18

19. Predicting Concurrency issues

20. public void synchronized getData() {
doSomething();
}
Thread 1
Thread 2
Thread 1
BLOCKED THREADS
Concurrency Problem
20

21. BLOCKED state threads
Micrometric
Source: Thread Dump

22. Case Study
Major Leisure Travel Service
Provider
https://blog.ycrash.io/2022/03/09/java-uuid-generation-performance-impact/
22

23. Predicting Poor Response Time

24. What is Garbage?
HTTP Request
Objects
Memory
Garbage
24

25. 25
3-4 Decades ago
Developer
Writes code to Manually evict Garbage
JVM
Automatically evicts Garbage
Now
How are objects Garbage Collected?
Evolution: Manual -> Automatic

26. 26
Automatic GC sounds good right?
Yes, but for
GC pauses CPU consumption

27. 27
Application suffering from Consecutive Full GCs

28. 28
Long GC Pause Duration

29. GC Pause Time
Micrometric
Source: Garbage Collection Log

30. Specific Errors/Exceptions
Micrometric
Source: Application Logs

31. My App
yCrash
agent
yCrash Server
Container/Machine
1
Every 3 minutes Micro-Metrics*
are captured
2 Metrics are transmitted
4 If problem forecasted,
360 ° data capture
is triggered
3 ML, Patterns applied on the Micro-Metrics
Cloud/On-premise
31
Micro-Metrics *
1. Garbage Collection Log
2. Thread Dump + top –H
3. Application Log
Micro-Metrics Monitoring Architecture

32. 1. GC Log
10. netstat
12. vmstat
2. Thread Dump
9. dmesg
3. Heap Dump
6. ps
8. Disk Usage
5. top 13. iostat
11. ping
14. Kernel Params
15. App Logs
16. metadata
4. Heap Substitute
7. top -H
32
Open-source script:
https://github.com/ycrash/yc-data-script
360° Troubleshooting artifacts

33. Ram Lakshmanan ram@tier1app.com
@tier1app https://www.linkedin.com/company/ycrash
This deck will be published in:
https://blog.ycrash.io
If you want to learn more …
33
THANK YOU
FRIENDS