This document summarizes a half-day tutorial on test design for fully automated build architectures presented by Melissa Benua of mParticle at STAREAST 2018. The tutorial covered guiding principles for test design including prioritizing important and reliable tests, structuring automated pipelines around components, packages, and releases, and monitoring test results through code coverage, flaky test handling, and logging versus counters. It also included exercises mapping test cases to functional boundaries and categories of tests to pipeline stages.

1. MG
Half-day Tutorials
Monday, April 30th, 2018
8:30 AM
Test Design for Fully Automated Build
Architecture
Presented by:
Melissa Benua
mParticle
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888---268---8770 ·· 904---278---0524 - info@techwell.com - http://www.stareast.techwell.com/

2. Melissa Benua
mParticle
Melissa Benua has worked in nearly every software development role—dev, test,
DevOps, and program management—at companies big and small. She's created
and run high-availability, high-quality services on products such as mParticle,
PlayFab, Bing, Cortana, and Xbox One. Melissa discovered her love of massively-
scaled systems while growing the Bing backend, where she honed the art of
keeping highly-available complex systems up while undergoing massive code
churn. Melissa isn’t afraid to mix traditional approaches with bold new ideas to
make her products better, faster, and more reliable. She’s passionate not only
about maximizing efficiency both in her product code and in her developer tools
but also about sharing best practices among colleagues and the tech world at
large.

3. 3/13/2018
1
1
Test Design for Fully
Automated Build
Architectures
Melissa Benua
Senior Technical Lead, mParticle
mbenua@gmail.com
@queenofcode
STAREAST 2018
2
About Me
Boeing Microsoft PlayFab mParticle

4. 3/13/2018
2
3
About The Tutorial
Source
Control
Track +
Code
Review
Build +
Test
Deploy
+
Monitor
4
Guiding Principles
What are we doing here?

5. 3/13/2018
3
5
G U I D I N G P R I N C I P L E S
Key Test Features
Important
Reliable
Specific
6
G U I D I N G P R I N C I P L E S
Importance
Triage scenarios based on priority
IMPOSSIBLE to cover every scenario
Understand what failures can be
tolerated
Run the most important tests first

6. 3/13/2018
4
7
G U I D I N G P R I N C I P L E S
Importance
Build Failure? UI Test Failure?
8
G U I D I N G P R I N C I P L E S
Reliability
No flakiness
No false-negatives or false-positives
Repeatable without human
intervention
Cleans up after itself

7. 3/13/2018
5
9
G U I D I N G P R I N C I P L E S
Reliability
10
G U I D I N G P R I N C I P L E S
Specificity
Clear answer to a clear question
Have one main goal
Don’t ‘boil the ocean’
Minimal overlapping coverage

8. 3/13/2018
6
11
G U I D I N G P R I N C I P L E S
Specificity
12
G U I D I N G P R I N C I P L E S
Proceed With Caution
Shared
resource
reliance
1
Single-
threading
2
Long duration
test time
3
Caching
4

9. 3/13/2018
7
13
G U I D I N G P R I N C I P L E S
Exercise: Test Cases
Photo
Gallery Site
UX FrontEnd
(HTML5 +
JS)
Logic
BackEnd
(Java)
Data Store
(NoSQL)
• Most important?
• Easiest?
• Most reliable?
14
Automated Pipeline Structure
What does it even look like?

10. 3/13/2018
8
15
A U T O M A T E D P I P E L I N E S T R U C T U R E
CI + CD Pipeline
Source
Control
Track +
Code
Review
Build +
Test
Deploy
+
Monitor
16
A U T O M A T E D P I P E L I N E S T R U C T U R E
Beyond Unit Tests
Develop: Diff Build
• Compile change against mainline
• Execute unit tests
Build: Continuous Integration
• Compile change as a part of mainline submit
• Execute functional tests
Deploy: Continuous Deployment
• Start staging environment
• Deploy staging environment
• Execute UI + load tests
• Project structure?
• Moving parts?
• Functional boundaries?
• Shared resources?
• Mocking potential?

11. 3/13/2018
9
17
A U T O M A T E D P I P E L I N E S T R U C T U R E
Example Service Architecture
UI Layer
RESTful API
Admin UI
Web Frontend
Backend
Layer
Auth Service
Logic Service
Cache Service
Data Layer
Database
File Storage
18

12. 3/13/2018
10
19
A U T O M A T E D P I P E L I N E S T R U C T U R E
The Matrix
Component
• Compile
• Unit
Package
• Functional
• Integration
Release
• Acceptance
• UX
• Load
Product
BE Machine
Function
Auth
Logic
FE Machine
Function
Web UI
Cache
20
A U T O M A T E D
End to End

13. 3/13/2018
11
21
A U T O M A T E D P I P E L I N E S T R U C T U R E
Exercise: Mapping Categories to Stages
Photo
Gallery Site
UX FrontEnd
(HTML5 +
JS)
Logic
BackEnd
(Java)
Data Store
(NoSQL)
• What should we run?
• When should we run it?
• How long should we wait?
22
A U T O M A T E D P I P E L I N E S T R U C T U R E
Key Takeaways
Easy to
conditionally run
different
categories
1
Find the balance
between too
many stages and
too few
2
Machine time is
MUCH cheaper
than human time
3
Fail fast and fail
often
4

14. 3/13/2018
12
23
Monitoring and Reporting
What just happened?
24
M O N I T O R I N G A N D R E P O R T I N G
Code Coverage
0
10
20
30
40
50
60
70
80
90
100
0 25 50 75 100
%ofBugsFound
Code Coverage %
Bugs vs Effort
% of Bugs Found
• Line vs branch coverage
• Reports and readability
• Data-driven decisions

15. 3/13/2018
13
25
M O N I T O R I N G A N D R E P O R T I N G
Flaky Test Handling
Cost vs Value
Net gain?
Test Failure
Fatigue
Ignoring failures?
Ease of
Detection
How do we know?
26
M O N I T O R I N G A N D R E P O R T I N G
Logging vs Counters
Text Log File? Graph?

16. 3/13/2018
14
27
M O N I T O R I N G A N D R E P O R T I N G
Exercise: What Goes Where?
Photo
Gallery Site
UX FrontEnd
(HTML5 +
JS)
Logic
BackEnd
(Java)
Data Store
(NoSQL)
28
P U T T I N G I T T O G E T H E R
Overall Summary
Follow natural
system
boundaries
1
Know where
your cutline is
and respect
your time
2
Don’t try to boil
the ocean
3
Use your data
wisely
4

17. 3/13/2018
15
29
Thank you!
Melissa Benua
mbenua@gmail.com
@queenofcode
http://www.queenofcode.net