We all know about the importance of automatic deployment of applications to production, but do we all know how to build an automated quality control system that continuously provides feedback on the application’s state and decides whether the application is ready for production rollout in terms of quality metrics? I have prepared a step-by-step algorithm for setting up such a system on your project, with specifics for web applications. I will talk about the best quality control practices at different levels, from linters to performance testing, from underwater problems to the ideal monitoring system.

1. A Tester's Guide to CI/CD
as an Automated Quality Control System

2. ALEXPSHE ALEXPSHE ALE
● SDET at JetBrains
● 20+ fully automated
quality control systems
● Big fan of process
automation 💚

3. Get an understanding
of the high-level algorithm
for building a quality system
Main takeaway
*There will be no technologies and specific solutions for individual cases here.

4. THE PROBLEM IS ..
Typical view on CI/CD and testing
Automation
tests
Not part of
CI/CD OR Automation
tests
Part of
CI/CD
Green/Red
NOT BAD, BUT…

5. CI/CD -> Automated Quality Control System
New feature developed
Quality Metrics measured
Responsible people notified
Next process started
Quality Gates called
Pipeline artifacts collected
Time to respond increased 34%
Compressed stored image
Allowed +10% Automessage to
#performance channel
Save node settings
and benchmark log
Request developer`s update

6. DELIVERY
PROCESS
For feature:
● Requirements testing
● Functional testing
● Non-functional testing
For product version:
● UAT testing
● Alpha/Beta testing
● Functional testing
● Non-functional testing
For release:
● Configuration testing
● Compatibility checks
● Availability checks
WHAT PROCESSES DO YOU NEED?

7. PROCESSES`S
MAP
New feature
New feature
New feature
UAT
Release

8. FOLLOW STEPS: FOLLOW STEPS:
● Write down ALL processes in your team
● Choose the most crucial, expensive and long
● Implement automatic testing control system

9. A TESTER`S GUIDE TO CI/CD
Preparation steps:
● Business Metrics
● Quality Metrics
● Quality Gates
Pipeline steps:
● Pipeline triggers
● Pipeline stages
● Pipeline criteria
Feedback steps:
● Metric`s collecting
● System notifications
● Call to next process
FOR EACH DELIVERY PROCESS ->
Follow steps

10. LET'S IMAGINE
WE ARE TIKTOK`s TESTERS

11. FEATURE ASSESSMENT
New feature
LET`S SETUP PIPELINE FOR

12. PREPARATION STEPS

13. PREPARATION
STEPS
QUALITY
SYSTEM
WHERE TO START?

14. PREPARATION
STEPS
1.BUSINESS
METRICS
WHY
Do we need
test automation?
Because:
We have to
decrease
manual testing
We have to
automate
test cases
We are
test automation
engineers

15. PREPARATION
STEPS
1.BUSINESS
METRICS
WHY
Do we need
test automation?
Business goal Business metrics Product metrics
More money More paid users +25% of retention
QA Auto as a tool to achieve product metrics
Then achieve
Then achieve

16. PREPARATION
STEPS
2.QUALITY
METRICS
Product metrics
Quality metrics should be
based on

17. PREPARATION
STEPS
LIST
OF
METRICS
LIST OF METRICS
Product metrics
Business Metric: Revenue Growth
Goal:
20% Overall Revenue Growth
for the Company
Over the Quarter
Product Metric: User Retention
Goal: User retention after first use
increases from 25% to 40% following
the implementation of:
feature that recommends content based
on the longest videos watched from the
last three
Business metrics

18. PREPARATION
STEPS
LIST
OF
METRICS
LIST OF METRICS Product metrics
Quality metrics
Quality Metric: Test Coverage
for New Recommendation Feature
Goal: Achieve 90%
test coverage for the new video
recommendation feature.
Product Metric: User Retention
Goal: User retention after first use
increases from 25% to 40% following
the implementation of:
feature that recommends content based
on the longest videos watched from the
last three

19. PREPARATION
STEPS
LIST
OF
METRICS
LIST OF METRICS
Quality metrics
FROM NON FUNCTIONAL REQUIREMENTS
Quality Metric: Performance
Goal: Ensure TikTok can
maintain smooth performance
during peak times, handling
up to 1 million concurrent
users without degradation in
service quality.

20. PREPARATION
STEPS
3.
QUALITY
GATES
based on Quality metrics
Quality gates

21. PREPARATION
STEPS
LIST
OF
METRICS
LIST OF METRICS Quality metrics
Quality gates
https://pshe-academy.com/quality-metrics/en
Quality Metric: Test Coverage
for New Recommendation Feature
Goal: Achieve 90%
test coverage for the new video
recommendation feature.
Quality Gates:
>= 90% - allowed
< 90% - not allowed

22. PREPARATION
STEPS
CONCLUSION
STEPS TO DO:
1. Ask for business goals for the next quater/year
2. Clarify product metrics with product owner
3. Build qulity metrics based on product metrics
4. Setup quality gates based on quality metrics

23. PIPELINE STEPS

24. PIPELINE
STEPS
1.
PIPELINE
TRIGGERS
Git events:
● New Merge Request
● New Commit
● New changes in folder
Manual actions:
● Manual start
● Person`s approve
● Message in Slack
Previous step ended:
● Triggered
by another pipeline
WHAT THE FLAG OF PROCESS START?

25. FEATURE ASSESSMENT
New feature
Triggered by commit:
1 commit = 1 MR
1. Limited resources
2. Have to get feedback before merge

26. PIPELINE
STEPS
2.
PIPELINE
STAGES
Functional testing:
● Unit tests
● Integration tests
● System tests
WHAT TYPES OF STAGES?
Code quality:
● Checkstyle
● Linters
Non-Functional testing:
● Performance tests
● Security tests
● Availability tests
Custom heuristics:
● Issue state
● Bug`s probability
Critical checks:
● Build
● Compilation
● Configuration

27. FEATURE ASSESSMENT
New feature
Triggered by commit:
1 commit = 1 MR
Build
Unit
tests
E2E
tests
Performance
Tests

28. PIPELINE
STEPS
3.
PIPELINE
CRITERIA
“FAIL FIRST” approach:
Pipeline has to fail
when it's obvious that
the process is not met
requirements
Time of execution:
Pipeline has to be finished
in appropriate time
(time that does not affect
other processes)
Reliability:
Pipeline has to provide
only true-positive feedback
(low flakiness rate)
WHAT CRITERIA OF THE EFFECTIVE PIPELINE?

29. FEATURE ASSESSMENT
New feature
Triggered by commit:
1 commit = 1 MR
Build
Unit
tests
E2E
tests
Performance
Tests
Optimize
build chain
Set Up
parallel runs
Increase
stability
Prepare
Prod-like env

30. PIPELINE
STEPS
CONCLUSION
STEPS TO DO:
1. Find correct trigger/s for the process
2. Design pipeline stages based on metrics/checks
3. Prioritize stages following “fail first” approach
4. Measure time/flakiness rate
to confirm pipeline quality criteria

31. FEEDBACK STEPS

32. FEEDBACK
STEPS
1.
METRIC`S
COLLECTING
Statistics from reports:
● Test Coverage
● Test Results
● History and metadata
Execution collectors:
● Retries
● Performance data
● Artifacts
Custom scripts:
● Issue status
● Deploy status
HOW TO COLLECT ALL METRICS?

33. FEATURE ASSESSMENT
New feature
Triggered by commit:
1 commit = 1 MR
Build
Unit
tests
E2E
tests
Performance
Tests
Build logs Test Report
Coverage Map
App logs
App performance metrics
Test Measurements

34. FEEDBACK
STEPS
1.
METRIC`S
COLLECTING
Ready-to-use solutions:
● Grafana
● Allure Reports
● Sonarqube
Custom Solutions:
● Notion Dashboard
● HTML report
● Data report tools
WHERE TO STORE ALL METRICS?

35. ALLURE REPORT
Test Coverage
for New
Recommendation Feature

36. GRAFANA
Availability checks

37. FEEDBACK
STEPS
2.
SYSTEM
NOTIFICATIONS
Step #1:
Level of
escalation
Step #2:
Notification
channels
HOW TO NOTIFY RESPONSIBLE PEOPLE?

38. FEATURE ASSESSMENT
New feature
Triggered by commit:
1 commit = 1 MR
Build
Unit
tests
E2E
tests
Performance
Tests
#build-chain
All crushes
#tiktok-dev
Not passed
#tiktok-qa
Coverage < 90%
#performance-team
Cannot handle 1m users
SLACK BOT

39. FEEDBACK
STEPS
3.
CALL
TO
THE
NEXT
PROCESS
CALL THE NEXT PROCESS
Automatically Manually

40. PROCESSES`S
MAP
New feature
New feature
New feature
UAT
Release
WHAT IS THE NEXT PROCESS?

41. FEATURE ASSESSMENT
New feature
LET`S SETUP PIPELINE FOR
AFTER
QA APPROVE

42. FEEDBACK
STEPS
CONCLUSION
STEPS TO DO:
1. Choose the best way to collect all metrics
2. Figure out responsibilities and call of actions
3. Setup a call to the next process based on
processes` map

43. AS A RESULT

44. AUTOMATED
QUALITY
CONTROL
SYSTEM
1. Prepared stage priority
2. Automated metrics calculation
3. Implemented instruction
4. Collecting data and call next step
CI/CD as tester`s tool
Continuous feedback about
system quality status

45. SLIDES
&
QUESTIONS
THANKS!
THANKS!
THAN