DevOpsDaysIndia2017 Talk by praveen shukla(product engineer at Go-Jek)

1. Reliability At Scale
Praveen Shukla

2. About Me
@_praveenshukla
@achilles42
Chaos Kid (Product Engineer)

3. - Transport, logistics, hyperlocal delivery and payments startup in
Indonesia
- 18 Products
- 40 Millions downloads
- 20 Millions Active users
- 300K driver in platform
- Top 50 Innovative Companies on fortune (go-jek @17)
- 1M TPS(internal API call)

4. - What is reliability ?
- Add simple solutions to achieve reliability when go-jek was small
- How we grew tremendously
- Problems while scaling
- How we came up with better and scalable solutions.
- Iteration
Agenda

5. Define “Reliable”

6. ● 4 nines
● MTBF
● Failure Per Year(AFR)
● QoS
● SLA

7. “FAILURES”
Systems operates outside specified
parameter.

8. Business: Users are complaining

9. Failure is subjective!

10. In 2015

11. Always a trade-off between
velocity and stability

12. ● CI / CD
● Configuration management
● Monitoring
● Alerting

13. ● CI / CD
● Configuration management
● Monitoring
● Alerting

14. ● CI / CD
● Configuration management
● Monitoring
● Alerting

15. ● CI / CD
● Configuration management
● Monitoring
● Alerting

16. ● CI / CD
● Configuration management
● Monitoring
● Alerting

18. In 2017

19. 2015
● 4 Products
● 10+ Microservices
● 100+ Instances
● 50+ Tech People
● 18 Products
● 250+ Microservices
● 8K+ Instances across 3
datacenters
● 350+ Tech People
4 X
25X
80X
7X
2017

21. ● Pipeline access management
● Custom Deployments
● DSL Repo Management
● No Branch Based Deployments
CI / CD

22. ● Every service has their own cookbook
● Cookbook dependency management
CM

23. ● Alert getting lost
● Not getting alerts to a right person
● Too many people getting too many pagers
● Who is responsible to take action on a particular alert
Monitoring & Alerting

24. Serious production outages
Business Loss

26. SOLUTIONS

27. ● Single place for Code, Build and Deploy
Access control
The CI Pipeline is just a yaml.
The CI file is part of the same source code.
Freedom to tweak their pipeline according to specific use case.
Provides us feature like branch and tag based deploys effortlessly
Gitlab and Gitlab-CI

28. Configuration Management

29. ● Master cookbooks concept
● Single cookbook to manage
● No. Of Stack == Number of Cookbooks

30. Smart Alert Router
● Every product has a group
● A group can have multiple microservices
● A microservices can have multiple servers
● One member can belongs to many groups

31. Architecture
Smart alert routerKapacitor
VM
Telegraf
Grafana
Alerting
TS Database
visualization
Agent on VM

33. Multiple Dependencies
R = 99 %

34. Multiple Dependencies
R = 96 %
R = 99 % R = 99 % R = 99 %

35. Multiple Dependencies
R = 88 %
R = 96 % R = 96 % R = 96 %
R = 99 % R = 99 % R = 99 % R = 99 % R = 99 %

36. Multiple Dependencies
R = 88 %
R = 96 % R = 96 % R = 96 %
R = 99 % R = 99 % R = 99 % R = 99 % R = 99 %
CIRCUIT BREAKERS!!

37. OPTION 1:

38. OPTION 2:

39. Really how does it affect your system !!
● 99.9930 = 99.7 % uptime
● 0.3% of 1 billion requests failing i.e 3,000,000 failed.
● 2+ hours downtime every month even after dependent
systems have excellent uptime.

40. Queuing Delay
p
Delay -------
1 - p
p = System Utilization

41. Queuing Delay
p
Delay -------
1 - p
p = System Utilization
THROTTLE YOUR SYSTEM!!

42. reliability is iterative process

43. THANKS!
questions?
_praveenshukla