SlideShare ist ein Scribd-Unternehmen logo
1 von 50
Downloaden Sie, um offline zu lesen
Log Management
System
for Microservices
Võ Duy Tuấn
Founder
Teamcrop.com
Vo Duy Tuan
 PHP, Javascript, Flutter
 Web Scalability & Optimization
 Data Mining
 ERP System
 Docker & Microservices
Interests
Founder @ Teamcrop.com
Why log management?
• System troubleshooting
• Application debugging
• Performance optimization
• Security
• Business decision making
Type of Logs
• Business
– Auditing
– Feature Usage
– Critical Function
• Operation
– Operating System
– General
– API Request
– SQL Query
– Distributed Tracing
Abstract Logging Design
Storage
Collecto
r
Dashboard
Syslog
PHP Log
Nginx Log
Access Log
Error Log
Activity Log
…
PHP
React JS
Grafana
File
Syslog
RabbitMQ
MySQL
Clickhouse
Anatomy of a Service @Teamcrop
Host (Linux)
Service Container (Docker)
Web Server (Nginx) + PHP
Application Request (eg: order)
SQL Query 1
SQL Query 2
Call another services request (Eg: customer, cashflow…)
OS log
Syslog
Access, Error log
API Request, Audit…
SQL Query log
Distributed tracing
| Business1
1.1. Auditing Log
• Metrics:
– Employee
– Service
– Section (controller)
– Action (add, update, delete, import..)
– Severity
– Change log
– IP Address
1.1. Store Auditing Log
Audit Log
Message
Queue
(RabbitMQ)
Queue Worker
(PHP)
Teamcrop API
(PHP)
Teamcrop Log
(MySQL)
1.1. View Auditing Log
Teamcrop
Audit Log API
(PHP)
Teamcrop Log
(MySQL)
Dashboard
(React JS)
Request Query
1.1. Auditing Log Stack
• Collector
– PHP
• Storage
– MySQL
• Dashboard
– React JS
1.1. Auditing Log Example
1.1. Auditing Log Example (cont.)
1.2. Feature Usage Log
• Metrics:
– Visitor
– Page View
– Time on Page
– Technology
– …
• Stack:
– Google Analytics
– Self-hosted application
1.2. Feature Usage Log Example
1.3. Critical Function Log
• Metrics:
– Raw POST data
• Stack:
– Collector: PHP
– Storage: File System
– Dashboard: React JS
1.3. Critical Function Log Example
| Operation2
2.1. Operating System Log
• Metrics:
– Memory (RAM)
– CPU
– Disk
– Network
– …
• Stack:
– NodeQuery.com
2.1. Operating System Log Example
2.1. Operating System Log Example
(cont.)
2.1. Operating System Log Example
(cont.)
2.2. General Log
• Metrics:
– PHP: error, warning…
– Nginx: access, error…
– Custom Log Message
– Cronjob Log
– Queue Log
– Others
2.2. Store General Log
General Log
(Access, Error Log)
Syslog
(syslog-ng)
UDP
UDP Log Server
(PHP)
Message
Queue
(RabbitMQ)
UDP Log Worker
(PHP)
Clickhouse
(Syslog Table)
2.2. View General Log
Teamcrop
Analytics API
(PHP)
Clickhouse
(Syslog Table)
Dashboard
(React JS)
Request Query
2.2. General Log Stack
• Collector:
– Syslog-ng
• Storage:
– RabbitMQ (Queue)
– Clickhouse
• Dashboard:
– PHP
– React JS
2.2. General Log Example
2.2. General Log Example (cont.)
2.3. API Request Log
• Metrics:
– HTTP Status
– HTTP Method
– Tenant
– Service name
– Function (controller, action)
– Execution time
– Memory
– Source
– …
2.3. Store API Request Log
API Request Log Syslog
(syslog-ng)
UDP
UDP Log Server
(PHP)
Message
Queue
(RabbitMQ)
UDP Log Worker
(PHP)Clickhouse
2.3. View API Request Log
Teamcrop
Analytics API
(PHP) Clickhouse
(API Request
Log Table)
Dashboard
(React JS)
Request Query
Grafana UI Grafana API
Request
Query
2.3. API Request Log Stack
• Collector:
– Syslog-ng
– Monolog (PHP)
• Storage:
– RabbitMQ (Queue)
– Clickhouse
• Dashboard:
– PHP
– React JS
– Grafana
2.3. API Request Log Example
2.3. API Request Log Example
(cont.)
2.3. API Request Log Example
(cont.)
2.3. API Request Log Example
(cont.)
2.4. SQL Query Log
• Metrics:
– Replication
– Table
– Query type (SELECT, INSERT…)
– Execution
– Tenant
– …
2.4. Store SQL Query Log
SQL Query Log Syslog
(syslog-ng)
UDP
UDP Log Server
(PHP)
Message
Queue
(RabbitMQ)
UDP Log Worker
(PHP)Clickhouse
2.4. View SQL Query Log
Teamcrop
Analytics API
(PHP) Clickhouse
(SQL Query
Log Table)
Dashboard
(React JS)
Request Query
Grafana UI Grafana API
Request
Query
2.4. SQL Query Log Stack
• Collector:
– Syslog-ng
– Monolog (PHP)
• Storage:
– RabbitMQ (Queue)
– Clickhouse
• Dashboard:
– PHP
– React JS
– Grafana
2.4. SQL Query Log Example
4. SQL Query Log Example (cont.)
4. SQL Query Log Example (cont.)
2.5. Distributed Tracing Log
• Metrics:
– Chain of requests
– Duration
– Execution
– Tenant
– Service
– …
2.5. Store Distributed Tracing Log
OpenTracing Log
(ZipKin format)
Syslog
(syslog-ng)
UDP
UDP Log Server
(PHP)
Message
Queue
(RabbitMQ)
UDP Log Worker
(PHP)
Jaeger
API
Jaeger
DB
2.5. Distributed Tracing Log Stack
• Collector:
– Syslog-ng
– Monolog (PHP)
– Zipkin Client Library (PHP)
• Storage:
– RabbitMQ (Queue)
– Jaeger
• Dashboard:
– Jaeger UI
2.5. Distributed Tracing Model
• OpenTracing Specification
2.5. Distributed Tracing Log Example
DEMO
RabbitMQ
Jeager UI
Clickhouse GUI Client
Customized Log Dashboard
NodeQuery
Grafana
Teamcrop Log Audit
Thanks!
CONTACT:
voduytuan@teamcrop.com
bloghoctap.com

Weitere ähnliche Inhalte

Was ist angesagt?

WSO2 API Manager 2.0 - Overview
WSO2 API Manager 2.0 - Overview WSO2 API Manager 2.0 - Overview
WSO2 API Manager 2.0 - Overview Edgar Silva
 
Asynchronous Processing with Outbox Pattern in .NET Core 3.0
Asynchronous Processing with Outbox Pattern in .NET Core 3.0Asynchronous Processing with Outbox Pattern in .NET Core 3.0
Asynchronous Processing with Outbox Pattern in .NET Core 3.0Baris Ceviz
 
CQRS and Event Sourcing for Java Developers
CQRS and Event Sourcing for Java DevelopersCQRS and Event Sourcing for Java Developers
CQRS and Event Sourcing for Java DevelopersMarkus Eisele
 
Api management best practices with wso2 api manager
Api management best practices with wso2 api managerApi management best practices with wso2 api manager
Api management best practices with wso2 api managerChanaka Fernando
 
Conhecendo Apache Kafka
Conhecendo Apache KafkaConhecendo Apache Kafka
Conhecendo Apache KafkaRafa Noronha
 
Domain Driven Design và Event Driven Architecture
Domain Driven Design và Event Driven Architecture Domain Driven Design và Event Driven Architecture
Domain Driven Design và Event Driven Architecture IT Expert Club
 
MongoDB vs. Postgres Benchmarks
MongoDB vs. Postgres Benchmarks MongoDB vs. Postgres Benchmarks
MongoDB vs. Postgres Benchmarks EDB
 
ASP.NET Core MVC + Web API with Overview
ASP.NET Core MVC + Web API with OverviewASP.NET Core MVC + Web API with Overview
ASP.NET Core MVC + Web API with OverviewShahed Chowdhuri
 
Building .NET Microservices
Building .NET MicroservicesBuilding .NET Microservices
Building .NET MicroservicesVMware Tanzu
 
Kubernetes
KubernetesKubernetes
Kuberneteserialc_w
 
Go micro framework to build microservices
Go micro framework to build microservicesGo micro framework to build microservices
Go micro framework to build microservicesTechMaster Vietnam
 

Was ist angesagt? (20)

devops
devops devops
devops
 
WSO2 API Manager 2.0 - Overview
WSO2 API Manager 2.0 - Overview WSO2 API Manager 2.0 - Overview
WSO2 API Manager 2.0 - Overview
 
Asynchronous Processing with Outbox Pattern in .NET Core 3.0
Asynchronous Processing with Outbox Pattern in .NET Core 3.0Asynchronous Processing with Outbox Pattern in .NET Core 3.0
Asynchronous Processing with Outbox Pattern in .NET Core 3.0
 
Introduction to Microservices
Introduction to MicroservicesIntroduction to Microservices
Introduction to Microservices
 
Introduction to Microservices
Introduction to MicroservicesIntroduction to Microservices
Introduction to Microservices
 
CQRS and Event Sourcing for Java Developers
CQRS and Event Sourcing for Java DevelopersCQRS and Event Sourcing for Java Developers
CQRS and Event Sourcing for Java Developers
 
Microservices
MicroservicesMicroservices
Microservices
 
contentful_sec
contentful_seccontentful_sec
contentful_sec
 
Why Microservice
Why Microservice Why Microservice
Why Microservice
 
Sapo Microservices Architecture
Sapo Microservices ArchitectureSapo Microservices Architecture
Sapo Microservices Architecture
 
Api management best practices with wso2 api manager
Api management best practices with wso2 api managerApi management best practices with wso2 api manager
Api management best practices with wso2 api manager
 
Conhecendo Apache Kafka
Conhecendo Apache KafkaConhecendo Apache Kafka
Conhecendo Apache Kafka
 
Domain Driven Design và Event Driven Architecture
Domain Driven Design và Event Driven Architecture Domain Driven Design và Event Driven Architecture
Domain Driven Design và Event Driven Architecture
 
MongoDB vs. Postgres Benchmarks
MongoDB vs. Postgres Benchmarks MongoDB vs. Postgres Benchmarks
MongoDB vs. Postgres Benchmarks
 
ASP.NET Core MVC + Web API with Overview
ASP.NET Core MVC + Web API with OverviewASP.NET Core MVC + Web API with Overview
ASP.NET Core MVC + Web API with Overview
 
Building .NET Microservices
Building .NET MicroservicesBuilding .NET Microservices
Building .NET Microservices
 
Introduction to Tekton
Introduction to TektonIntroduction to Tekton
Introduction to Tekton
 
Architecture: Microservices
Architecture: MicroservicesArchitecture: Microservices
Architecture: Microservices
 
Kubernetes
KubernetesKubernetes
Kubernetes
 
Go micro framework to build microservices
Go micro framework to build microservicesGo micro framework to build microservices
Go micro framework to build microservices
 

Ähnlich wie Log management system for Microservices

Extreme Replication - Performance Tuning Oracle GoldenGate
Extreme Replication - Performance Tuning Oracle GoldenGateExtreme Replication - Performance Tuning Oracle GoldenGate
Extreme Replication - Performance Tuning Oracle GoldenGateBobby Curtis
 
Log Data Analysis Platform by Valentin Kropov
Log Data Analysis Platform by Valentin KropovLog Data Analysis Platform by Valentin Kropov
Log Data Analysis Platform by Valentin KropovSoftServe
 
Log Data Analysis Platform
Log Data Analysis PlatformLog Data Analysis Platform
Log Data Analysis PlatformValentin Kropov
 
Triangle Node.js DevOps
Triangle Node.js DevOpsTriangle Node.js DevOps
Triangle Node.js DevOpsShubhra Kar
 
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly SolarWinds Loggly
 
VMworld 2013: Deep Dive into vSphere Log Management with vCenter Log Insight
VMworld 2013: Deep Dive into vSphere Log Management with vCenter Log InsightVMworld 2013: Deep Dive into vSphere Log Management with vCenter Log Insight
VMworld 2013: Deep Dive into vSphere Log Management with vCenter Log InsightVMworld
 
Tuning Elasticsearch Indexing Pipeline for Logs
Tuning Elasticsearch Indexing Pipeline for LogsTuning Elasticsearch Indexing Pipeline for Logs
Tuning Elasticsearch Indexing Pipeline for LogsSematext Group, Inc.
 
Dealing with and learning from the sandbox
Dealing with and learning from the sandboxDealing with and learning from the sandbox
Dealing with and learning from the sandboxElaine Van Bergen
 
Transforming Mobile Push Notifications with Big Data
Transforming Mobile Push Notifications with Big DataTransforming Mobile Push Notifications with Big Data
Transforming Mobile Push Notifications with Big Dataplumbee
 
How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.Renzo Tomà
 
Cloud Foundry Monitoring How-To: Collecting Metrics and Logs
Cloud Foundry Monitoring How-To: Collecting Metrics and LogsCloud Foundry Monitoring How-To: Collecting Metrics and Logs
Cloud Foundry Monitoring How-To: Collecting Metrics and LogsAltoros
 
Meetup milano #4 log management and anypoint advanced monitoring
Meetup milano #4   log management and anypoint advanced monitoringMeetup milano #4   log management and anypoint advanced monitoring
Meetup milano #4 log management and anypoint advanced monitoringGonzalo Marcos Ansoain
 
Dealing with and learning from the sandbox
Dealing with and learning from the sandboxDealing with and learning from the sandbox
Dealing with and learning from the sandboxElaine Van Bergen
 
Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...
Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...
Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...Amazon Web Services
 
VMworld 2013: Performance Management of Business Critical Applications using ...
VMworld 2013: Performance Management of Business Critical Applications using ...VMworld 2013: Performance Management of Business Critical Applications using ...
VMworld 2013: Performance Management of Business Critical Applications using ...VMworld
 
Log aggregation and analysis
Log aggregation and analysisLog aggregation and analysis
Log aggregation and analysisDhaval Mehta
 
NoSQL em Windows Azure Table Storage - Vitor Tomaz
NoSQL em Windows Azure Table Storage - Vitor TomazNoSQL em Windows Azure Table Storage - Vitor Tomaz
NoSQL em Windows Azure Table Storage - Vitor TomazComunidade NetPonto
 
[NetPonto] NoSQL em Windows Azure Table Storage
[NetPonto] NoSQL em Windows Azure Table Storage[NetPonto] NoSQL em Windows Azure Table Storage
[NetPonto] NoSQL em Windows Azure Table StorageVitor Tomaz
 

Ähnlich wie Log management system for Microservices (20)

Extreme Replication - Performance Tuning Oracle GoldenGate
Extreme Replication - Performance Tuning Oracle GoldenGateExtreme Replication - Performance Tuning Oracle GoldenGate
Extreme Replication - Performance Tuning Oracle GoldenGate
 
Log Data Analysis Platform by Valentin Kropov
Log Data Analysis Platform by Valentin KropovLog Data Analysis Platform by Valentin Kropov
Log Data Analysis Platform by Valentin Kropov
 
Log Data Analysis Platform
Log Data Analysis PlatformLog Data Analysis Platform
Log Data Analysis Platform
 
Triangle Node.js DevOps
Triangle Node.js DevOpsTriangle Node.js DevOps
Triangle Node.js DevOps
 
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
 
Docker Logging Webinar
Docker Logging  WebinarDocker Logging  Webinar
Docker Logging Webinar
 
VMworld 2013: Deep Dive into vSphere Log Management with vCenter Log Insight
VMworld 2013: Deep Dive into vSphere Log Management with vCenter Log InsightVMworld 2013: Deep Dive into vSphere Log Management with vCenter Log Insight
VMworld 2013: Deep Dive into vSphere Log Management with vCenter Log Insight
 
Tuning Elasticsearch Indexing Pipeline for Logs
Tuning Elasticsearch Indexing Pipeline for LogsTuning Elasticsearch Indexing Pipeline for Logs
Tuning Elasticsearch Indexing Pipeline for Logs
 
Log Files
Log FilesLog Files
Log Files
 
Dealing with and learning from the sandbox
Dealing with and learning from the sandboxDealing with and learning from the sandbox
Dealing with and learning from the sandbox
 
Transforming Mobile Push Notifications with Big Data
Transforming Mobile Push Notifications with Big DataTransforming Mobile Push Notifications with Big Data
Transforming Mobile Push Notifications with Big Data
 
How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.
 
Cloud Foundry Monitoring How-To: Collecting Metrics and Logs
Cloud Foundry Monitoring How-To: Collecting Metrics and LogsCloud Foundry Monitoring How-To: Collecting Metrics and Logs
Cloud Foundry Monitoring How-To: Collecting Metrics and Logs
 
Meetup milano #4 log management and anypoint advanced monitoring
Meetup milano #4   log management and anypoint advanced monitoringMeetup milano #4   log management and anypoint advanced monitoring
Meetup milano #4 log management and anypoint advanced monitoring
 
Dealing with and learning from the sandbox
Dealing with and learning from the sandboxDealing with and learning from the sandbox
Dealing with and learning from the sandbox
 
Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...
Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...
Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...
 
VMworld 2013: Performance Management of Business Critical Applications using ...
VMworld 2013: Performance Management of Business Critical Applications using ...VMworld 2013: Performance Management of Business Critical Applications using ...
VMworld 2013: Performance Management of Business Critical Applications using ...
 
Log aggregation and analysis
Log aggregation and analysisLog aggregation and analysis
Log aggregation and analysis
 
NoSQL em Windows Azure Table Storage - Vitor Tomaz
NoSQL em Windows Azure Table Storage - Vitor TomazNoSQL em Windows Azure Table Storage - Vitor Tomaz
NoSQL em Windows Azure Table Storage - Vitor Tomaz
 
[NetPonto] NoSQL em Windows Azure Table Storage
[NetPonto] NoSQL em Windows Azure Table Storage[NetPonto] NoSQL em Windows Azure Table Storage
[NetPonto] NoSQL em Windows Azure Table Storage
 

Mehr von Võ Duy Tuấn

Multi-tenant Database Design for SaaS
Multi-tenant Database Design for SaaSMulti-tenant Database Design for SaaS
Multi-tenant Database Design for SaaSVõ Duy Tuấn
 
Mobile outsourcing best practices
Mobile outsourcing best practicesMobile outsourcing best practices
Mobile outsourcing best practicesVõ Duy Tuấn
 
Chatbot in Sale Management
Chatbot in Sale ManagementChatbot in Sale Management
Chatbot in Sale ManagementVõ Duy Tuấn
 
Microservices and docker
Microservices and dockerMicroservices and docker
Microservices and dockerVõ Duy Tuấn
 
Scale with Microservices
Scale with MicroservicesScale with Microservices
Scale with MicroservicesVõ Duy Tuấn
 
Microservices in production
Microservices in productionMicroservices in production
Microservices in productionVõ Duy Tuấn
 
Business Intelligence in Retail Industry
Business Intelligence in Retail IndustryBusiness Intelligence in Retail Industry
Business Intelligence in Retail IndustryVõ Duy Tuấn
 
How to Build Recommender System with Content based Filtering
How to Build Recommender System with Content based FilteringHow to Build Recommender System with Content based Filtering
How to Build Recommender System with Content based FilteringVõ Duy Tuấn
 
Php psr standard 2014 01-22
Php psr standard 2014 01-22Php psr standard 2014 01-22
Php psr standard 2014 01-22Võ Duy Tuấn
 
Speed up with hiphop php 2014 01-22
Speed up with hiphop php 2014 01-22Speed up with hiphop php 2014 01-22
Speed up with hiphop php 2014 01-22Võ Duy Tuấn
 
How to build a Recommender System
How to build a Recommender SystemHow to build a Recommender System
How to build a Recommender SystemVõ Duy Tuấn
 
Reader.vn 2012 - The Book Of Life
Reader.vn 2012 - The Book Of LifeReader.vn 2012 - The Book Of Life
Reader.vn 2012 - The Book Of LifeVõ Duy Tuấn
 
Heavy Web Optimization: Backend
Heavy Web Optimization: BackendHeavy Web Optimization: Backend
Heavy Web Optimization: BackendVõ Duy Tuấn
 
Heavy Web Optimization: Frontend
Heavy Web Optimization: FrontendHeavy Web Optimization: Frontend
Heavy Web Optimization: FrontendVõ Duy Tuấn
 
Caching strategy and apc
Caching strategy and apcCaching strategy and apc
Caching strategy and apcVõ Duy Tuấn
 
PHP: Debugger, Profiler and more
PHP: Debugger, Profiler and morePHP: Debugger, Profiler and more
PHP: Debugger, Profiler and moreVõ Duy Tuấn
 
Magento overview and how sell Magento extensions
Magento overview and how sell Magento extensionsMagento overview and how sell Magento extensions
Magento overview and how sell Magento extensionsVõ Duy Tuấn
 

Mehr von Võ Duy Tuấn (20)

Multi-tenant Database Design for SaaS
Multi-tenant Database Design for SaaSMulti-tenant Database Design for SaaS
Multi-tenant Database Design for SaaS
 
Flutter introduction
Flutter introductionFlutter introduction
Flutter introduction
 
Mobile outsourcing best practices
Mobile outsourcing best practicesMobile outsourcing best practices
Mobile outsourcing best practices
 
Chatbot in Sale Management
Chatbot in Sale ManagementChatbot in Sale Management
Chatbot in Sale Management
 
Microservices and docker
Microservices and dockerMicroservices and docker
Microservices and docker
 
Scale with Microservices
Scale with MicroservicesScale with Microservices
Scale with Microservices
 
React introduction
React introductionReact introduction
React introduction
 
Microservices in production
Microservices in productionMicroservices in production
Microservices in production
 
Business Intelligence in Retail Industry
Business Intelligence in Retail IndustryBusiness Intelligence in Retail Industry
Business Intelligence in Retail Industry
 
How to Build Recommender System with Content based Filtering
How to Build Recommender System with Content based FilteringHow to Build Recommender System with Content based Filtering
How to Build Recommender System with Content based Filtering
 
Php psr standard 2014 01-22
Php psr standard 2014 01-22Php psr standard 2014 01-22
Php psr standard 2014 01-22
 
Speed up with hiphop php 2014 01-22
Speed up with hiphop php 2014 01-22Speed up with hiphop php 2014 01-22
Speed up with hiphop php 2014 01-22
 
How to build a Recommender System
How to build a Recommender SystemHow to build a Recommender System
How to build a Recommender System
 
Mobile for web
Mobile for webMobile for web
Mobile for web
 
Reader.vn 2012 - The Book Of Life
Reader.vn 2012 - The Book Of LifeReader.vn 2012 - The Book Of Life
Reader.vn 2012 - The Book Of Life
 
Heavy Web Optimization: Backend
Heavy Web Optimization: BackendHeavy Web Optimization: Backend
Heavy Web Optimization: Backend
 
Heavy Web Optimization: Frontend
Heavy Web Optimization: FrontendHeavy Web Optimization: Frontend
Heavy Web Optimization: Frontend
 
Caching strategy and apc
Caching strategy and apcCaching strategy and apc
Caching strategy and apc
 
PHP: Debugger, Profiler and more
PHP: Debugger, Profiler and morePHP: Debugger, Profiler and more
PHP: Debugger, Profiler and more
 
Magento overview and how sell Magento extensions
Magento overview and how sell Magento extensionsMagento overview and how sell Magento extensions
Magento overview and how sell Magento extensions
 

Kürzlich hochgeladen

YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.JasonViviers2
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introductionsanjaymuralee1
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationGiorgio Carbone
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024Becky Burwell
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Guido X Jansen
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxVenkatasubramani13
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best PracticesDataArchiva
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityAggregage
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?sonikadigital1
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...PrithaVashisht1
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptaigil2
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerPavel Šabatka
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionajayrajaganeshkayala
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructuresonikadigital1
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxDwiAyuSitiHartinah
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Vladislav Solodkiy
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)Data & Analytics Magazin
 

Kürzlich hochgeladen (17)

YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introduction
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - Presentation
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptx
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .ppt
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayer
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual intervention
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructure
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)
 

Log management system for Microservices

  • 1. Log Management System for Microservices Võ Duy Tuấn Founder Teamcrop.com
  • 2. Vo Duy Tuan  PHP, Javascript, Flutter  Web Scalability & Optimization  Data Mining  ERP System  Docker & Microservices Interests Founder @ Teamcrop.com
  • 3. Why log management? • System troubleshooting • Application debugging • Performance optimization • Security • Business decision making
  • 4. Type of Logs • Business – Auditing – Feature Usage – Critical Function • Operation – Operating System – General – API Request – SQL Query – Distributed Tracing
  • 5. Abstract Logging Design Storage Collecto r Dashboard Syslog PHP Log Nginx Log Access Log Error Log Activity Log … PHP React JS Grafana File Syslog RabbitMQ MySQL Clickhouse
  • 6. Anatomy of a Service @Teamcrop Host (Linux) Service Container (Docker) Web Server (Nginx) + PHP Application Request (eg: order) SQL Query 1 SQL Query 2 Call another services request (Eg: customer, cashflow…) OS log Syslog Access, Error log API Request, Audit… SQL Query log Distributed tracing
  • 8. 1.1. Auditing Log • Metrics: – Employee – Service – Section (controller) – Action (add, update, delete, import..) – Severity – Change log – IP Address
  • 9. 1.1. Store Auditing Log Audit Log Message Queue (RabbitMQ) Queue Worker (PHP) Teamcrop API (PHP) Teamcrop Log (MySQL)
  • 10. 1.1. View Auditing Log Teamcrop Audit Log API (PHP) Teamcrop Log (MySQL) Dashboard (React JS) Request Query
  • 11. 1.1. Auditing Log Stack • Collector – PHP • Storage – MySQL • Dashboard – React JS
  • 12. 1.1. Auditing Log Example
  • 13. 1.1. Auditing Log Example (cont.)
  • 14. 1.2. Feature Usage Log • Metrics: – Visitor – Page View – Time on Page – Technology – … • Stack: – Google Analytics – Self-hosted application
  • 15. 1.2. Feature Usage Log Example
  • 16. 1.3. Critical Function Log • Metrics: – Raw POST data • Stack: – Collector: PHP – Storage: File System – Dashboard: React JS
  • 17. 1.3. Critical Function Log Example
  • 19. 2.1. Operating System Log • Metrics: – Memory (RAM) – CPU – Disk – Network – … • Stack: – NodeQuery.com
  • 20. 2.1. Operating System Log Example
  • 21. 2.1. Operating System Log Example (cont.)
  • 22. 2.1. Operating System Log Example (cont.)
  • 23. 2.2. General Log • Metrics: – PHP: error, warning… – Nginx: access, error… – Custom Log Message – Cronjob Log – Queue Log – Others
  • 24. 2.2. Store General Log General Log (Access, Error Log) Syslog (syslog-ng) UDP UDP Log Server (PHP) Message Queue (RabbitMQ) UDP Log Worker (PHP) Clickhouse (Syslog Table)
  • 25. 2.2. View General Log Teamcrop Analytics API (PHP) Clickhouse (Syslog Table) Dashboard (React JS) Request Query
  • 26. 2.2. General Log Stack • Collector: – Syslog-ng • Storage: – RabbitMQ (Queue) – Clickhouse • Dashboard: – PHP – React JS
  • 27. 2.2. General Log Example
  • 28. 2.2. General Log Example (cont.)
  • 29. 2.3. API Request Log • Metrics: – HTTP Status – HTTP Method – Tenant – Service name – Function (controller, action) – Execution time – Memory – Source – …
  • 30. 2.3. Store API Request Log API Request Log Syslog (syslog-ng) UDP UDP Log Server (PHP) Message Queue (RabbitMQ) UDP Log Worker (PHP)Clickhouse
  • 31. 2.3. View API Request Log Teamcrop Analytics API (PHP) Clickhouse (API Request Log Table) Dashboard (React JS) Request Query Grafana UI Grafana API Request Query
  • 32. 2.3. API Request Log Stack • Collector: – Syslog-ng – Monolog (PHP) • Storage: – RabbitMQ (Queue) – Clickhouse • Dashboard: – PHP – React JS – Grafana
  • 33. 2.3. API Request Log Example
  • 34. 2.3. API Request Log Example (cont.)
  • 35. 2.3. API Request Log Example (cont.)
  • 36. 2.3. API Request Log Example (cont.)
  • 37. 2.4. SQL Query Log • Metrics: – Replication – Table – Query type (SELECT, INSERT…) – Execution – Tenant – …
  • 38. 2.4. Store SQL Query Log SQL Query Log Syslog (syslog-ng) UDP UDP Log Server (PHP) Message Queue (RabbitMQ) UDP Log Worker (PHP)Clickhouse
  • 39. 2.4. View SQL Query Log Teamcrop Analytics API (PHP) Clickhouse (SQL Query Log Table) Dashboard (React JS) Request Query Grafana UI Grafana API Request Query
  • 40. 2.4. SQL Query Log Stack • Collector: – Syslog-ng – Monolog (PHP) • Storage: – RabbitMQ (Queue) – Clickhouse • Dashboard: – PHP – React JS – Grafana
  • 41. 2.4. SQL Query Log Example
  • 42. 4. SQL Query Log Example (cont.)
  • 43. 4. SQL Query Log Example (cont.)
  • 44. 2.5. Distributed Tracing Log • Metrics: – Chain of requests – Duration – Execution – Tenant – Service – …
  • 45. 2.5. Store Distributed Tracing Log OpenTracing Log (ZipKin format) Syslog (syslog-ng) UDP UDP Log Server (PHP) Message Queue (RabbitMQ) UDP Log Worker (PHP) Jaeger API Jaeger DB
  • 46. 2.5. Distributed Tracing Log Stack • Collector: – Syslog-ng – Monolog (PHP) – Zipkin Client Library (PHP) • Storage: – RabbitMQ (Queue) – Jaeger • Dashboard: – Jaeger UI
  • 47. 2.5. Distributed Tracing Model • OpenTracing Specification
  • 49. DEMO RabbitMQ Jeager UI Clickhouse GUI Client Customized Log Dashboard NodeQuery Grafana Teamcrop Log Audit