Lisbon Mulesoft Meetup about Logging Aggregation & Visualization

1. 23/02/2020
Lisbon MuleSoft Meetup Group
LOGGING AGGREGATION & VISUALIZATION

2. 2
● Introductions
● Logging Aggregation - Fidelidade
● Demo + Q&A
● JSON Logger – Timestamp:its
● Demo + Q&A
● Quiz Competition: GET A VOUCHER !
Agenda

3. 3
●About the organizer:
○ Meetup Leader: Steve Fernandes
○ Meetup Speaker: Bruno Silva
○ Meetup Speaker: Tiago Miguens
●Sponsors:
○ Fidelidade
○ Timestamp:its
Introductions
A SHOW OF HANDS:
Who is new to this Meetup?

4. @ Fidelidade
LOGGING AGGREGATION &
VISUALIZATION

5. Fidelidade Overview

6. 6
Fidelidade Group Overview
National Operation & Head Quarter
• FIDELIDADE (PORTUGAL)
Internacional Operations
• FIDELIDADE SPAIN
• FIDELIDADE FRANCE
• GARANTIA (CAPE VERDE)
• FIDELIDADE ANGOLA
• FIDELIDADE MOZAMBIQUE
• FIDELIDADE MACAO
• LA POSITIVA (PERU)
• ALIANZA (BOLIVIA)
• ALIANZA GARANTIA (PARAGUAY)
• FIDELIDADE CHILE
Portugal Spain France Cape Verde Angola Mozambique Macao Peru Chile Bolivia Paraguay

7. Fidelidade IT Overview
Since 2018, Fidelidade IT has adopted a
more Agile structure, organized by
Clusters
7

8. 8
Fidelidade SOA Architecture
MuleSoft
MuleSoft
Public Portal
B2B Channels
Outsystems
B2E Channels
Outsystems
B2C Channels
Outsystems/Sharepoint
Contact Center
OneContact
Financial
SAP
Business Process Management
eFlow
Document Management
Centera
Content Management
Sharepoint
Business Intelligence
SAS
Operational Systems
zSeries/iSeries
Cloud Services
Azure IoT/Google
Fidelidade SOA
global architecture
provides
integration
capabilities
fundamental for the
different business
models
Note: not exhaustive
The architecture of Fidelidade is based on a SOA model allowing to take advantage of the
core systems and the growth of Digital

9. Technology Shift With Mulesoft

10. 10
Architecture Before Mulesoft
● A myriad of different platforms and products
○ Some for the same purpose
○ Outdated
● Weak REST Support
● Service catalog not properly integrated
Frontends
Channels
IBM Mainframe,
Databases, …
B2B
…
External
Access
B2B
(.NET)
Oracle API
Gateway
Integration
Layer
MTOM
(.NET)
SMP
(.NET)
Oracle
Service Bus
Business Service Gateway (.NET)
(Logic, Orquestration & Broker Services
Backends
Support
Tools
Oracle
Enterprise
Repository
Log
Viewer
(custom)
External
Partners
…
Outsystems Apps, SAP, BSG Apps, SAS, BPM,
Document Management, …
LoB
Apps
Mobile
Apps
External
Web
Apps
Internal
Web
Apps
Wearable
s
SOA
P
SOA
P
SOAP/RE
ST
SOA
P
SOA
P
SOA
P
SOAP/RE
ST
SOAP/RE
ST
SOAP/RE
ST
SOAP/RE
ST

11. 11
Challenge: Evolve Logging Model
● Custom Outsystems Application for
logging serving SMP and OSB
● Payload Logging (requests and responses)
● SOAP/XML Oriented.
JSON payloads represented in XML
● No correlation between requests
● Searchable fields:
○ Method
○ Date
○ User /Frontend User
○ Client Application
○ Frontend Session ID
○ Integration Server
○ Errored Requests
Error Details

12. 12
Driver For Change
● Reduce Technology Stack
● Higher level of integration capabilities.
○ Improve performance, availability, scalability and stability
○ Support for emerging technologies.
● Better alignment with market trends:
○ Restful API.
○ Micro services.
● Cloud & On-Premise architectures transparency
● API marketplace that merges all API and integrations under the same lifecycle and governance model.
● Collect, process and communicate monitoring information in real-time.

13. 13
Target Architecture
● Single Integration Platform
○ Private API Runtimes (On-prem)
○ Public API Runtime (Azure Cloud)
● Restful APIS.
● Some processes ensured by Mulesoft following
micro-services approach.
● Anypoint Platform as a unified platform for
designing, documenting, managing and
monitoring.
● Logging and Monitoring capabilities with Elastic
Stack (Filebeat, Elastic and Kibana)
Frontends
Channels
External
Access
Integration
Layer
Business Service Gateway (.NET)
(Complex Logic & Broker Services
SOAP
/REST
Support
Tools
MuleSoft
Public API Runtimes
MuleSoft
Private API Runtimes
MuleSoft
Control Plane
(Anypoint Platform)
E
L
K
IBM Mainframe, Databases,
…
…
Backends
Outsystems Apps, SAP, BSG Apps, SAS, BPM,
Document Management, …
LoB
Apps
…
External
Partners
Mobile
Apps
External
Web
Apps
Internal
Web
Apps
Wearable
s
SOAP/RE
ST
SOAP/RE
ST
SOAP/RE
ST
REST
REST
Micro
Serviços
REST

14. @ Fidelidade
Logging Framework

15. 15
Log Writing Overview
Mule
Server
MuleSoft
Private API Runtimes
Log4j on each
Mule Application with
Async Logger
Tracepoints written in
log files with log4j during
executions
Log Entry for each
tracepoint
Log data
API_1 API_2
API_3 API_4
API_1.log
API_2.log
API_3.log
API_4.log

16. 16
Log Data Structure– Main Fields
Log Step
Log tracepoint. Possible Values:
•SERVICE_START
•SERVICE_END
•SERVICE_ERROR
•ADAPTER_START
•ADAPTER_END
•ADAPTER_ERROR
•DEBUG
Integration ID Unique identifier for the API request. Shared between all log tracepoints within the same mule
application execution.
Correlation ID Identifier that correlate the API request with others within an orchestration. Shared between all log
tracepoints in any mule application execution in an orchestration.
Application
Name
Name of the Studio project/Mule Application
Location Request path that was called. In SOAP requests is suffixed by the SOAP operation.
Log TimeStamp Timestamp for the tracepoint.
Status Code Error code or success code. Applies to *_END or *_ERROR tracepoints. In SERVICE_END or
SERVICE_ERROR is the returned HTTP Status Code.
Status Message Error detailed description or success message.
Engine Machine
Name
Mule runtime hostname with listener port.
Payload
JSON Object that contains the mule event data processed by the Mule application. Is comprised of :
• payload - serialized string of the text, json or xml object.
• headers – JSON Object
• uriParams– JSON Object
• queryParams– JSON Object
In the case of an ADAPTER_ERROR the error description will be printed.
This attribute is configurable based on the log level for the mule application, which defines the amount
of information available. (Explained later)
Execution
Miliseconds
The duration of the scope (Service or Adapter) in milliseconds.
This field is only relevant in *_END or *_ERROR tracepoints.
json format

17. 17
Log Connector
● Custom Logger Connector with
Operations for each tracepoint
● Referenced in pom.xml in every mule project along
with other foundational assets .
● Maven plugin used in Jenkins CI/CD Pipelines
guarantees that the newest version is always used.
● Supported by a template to
facilitate developers.

18. 18
Connector Usage and Tracepoints
● Init Flow – Custom Component (Flow) reused by
all mule projects that sets variables used by logger
connector.
● Service Start – Logs the request received.
● Adapter – Scope that writes the following
tracepoints:
○ Adapter Start – Logs the request sent to backend
service.
○ Adapter End - Logs the response returned by the
backend service.
○ Adapter Error – Logs the error that occurred when
invoking the backend service.
● Service End – Logs the response returned.
● Error Handler Flow – Custom Component (Flow)
reused by all mule projects that builds the
common error structure and sets variables used
by logger connector
● Service Error – Logs the error response
returned.
● Debug – Used when an additional tracepoint is

19. 19
Connector Development
Non-Bocking Execution
Model
Connector parameters with
default values and GUI definition
Additional Capabilities
(e.g. obfuscation)
Tracepoints log writing
(log4j)
5 ms per
tracepoint
20 ms total

20. 20
Log Levels
● Log Levels – Strategy to reduce size footprint in production (> 10 M daily transactions)
○ DEBUG MODE (Default in Non Productive)
■ All tracepoints
■ With payload
○ REGULAR MODE (Default in Productive)
■ Success execution
● Only SERVICE_END tracepoint
● No Payload
■ Functional or Technical Error
● SERVICE_START tracepoint
● SERVICE_ERROR tracepoint
● With Payload
● Defined in Runtime Manager
○ Settings > Logging
● Automatic reset of default log levels for all applications (triggered by Jenkins on a regular basis)

21. 21
Other Capabilities
● Translation of client_id
○ In memory object store that contains client_id / Client Application Name pairs.
○ Refreshed, when necessary, with information retrieved after invoking Anypoint
Platform REST API
● Obfuscation
○ Sensible or large payload fields or headers (e.g. credentials, base64 fields, etc.)
■ JSON or XML
○ Defined per mule application through properties.

22. @ Fidelidade
Aggregation and Visualization
With ELK

23. 23
ELK Stack
● Beats (Collection) – Agent (Filebeat) installed in every mule server
machine
○ Competes for resources (memory, CPU, bandwith) with mule runtime
JVM.
○ Monitors changes in log files and sends them to Logstash.
● Logstash (Filtering and Aggregation) - Parse, filter and
transform.
○ Uses GROK Expressions to build JSON events with log data
○ Runs on dedicated Elastic servers.
● Elastic Search (Storage and Indexing) – Nonrelation DB for fast index and
search.
○ Indexes information.
○ Runs on dedicated Elastic servers. 3-Machine Cluster in Production
○ 1 month retention period in Production. 1 week in other environments.
● Kibana (Analysis and Visualization) – GUI to visualize the content indexed in
Elasticsearch cluster

24. 24
Discover - Querying in Kibana
Space /Environment
Query (KQL)
Time Interval
Selection
Log Tracepoints
(ELK Document)
Counter Chart

25. Discover - Querying in Kibana (Cont.)
Filtering and Column Selection
Table View
25

26. Correlation In Practice
Filtering by CorrelationId
Orchestrated requests
26

27. Log Data (Payload, Headers and Errors)
● DEBUG Mode
● REGULAR Mode
● Adapter Error Detail
Payload Available
Obfuscated data
client_id translation
Payload Available in
successful executions
27

28. Dashboards
28
● Custom views over elasticsearch data to address specific visualization needs.
Pulls together different visualization components (e.g. charts, filters, data tables)

29. Dashboard – Example 1 – Global Vision
29
● Focuses on total API invocations and performance

30. Dashboard – Example 2 – API with Errors
30
● Focuses on errors

31. Dashboard – Example 3 – Consumers
31
● Focuses on client applications (API Consumers)

32. DEMO

33. Generic Logging Solution
@ Timestamp:its

34. Structured Logging
Why it’s important to have a good logging structure?
● Enforce standard data to be logged in all scenarios for better operations
● Produce logs that are human and machine readable and in standard format
● Parse logs to provide a rich data structure instead of a string message (searchable)
● Build reports and dashboards around log data such as errors, exceptions and business events
34

35. Mulesoft JSON Logger
● Open Source (free) by Mulesoft Professional Services
● Published into your private exchange
● Provides a base set of recommended fields as well as message field
● Direct integration to Anypoint MQ
● Focus on Performance - Community supported
● Used worldwide, multiple customers
35
https://github.com/mulesoft-consulting/json-logger

36. Mulesoft JSON Logger - Configuration
36

37. Mulesoft JSON Logger – Component variables
37

38. Example of JSON Logger
38

39. 39

40. How to Integrate ELK with MuleSoft
40

41. Kibana - Discover
41

42. DEMO

43. Networking time
Introduce yourself to your neighbor

44. Thank you