Keynote presentation at DevOps Summit 2016 in Taipei.
http://devopssummit.ithome.com.tw/
I explained the importance of Test Automation and Continuous Integration for cultural change in DevOps context.
DevOpsのコンテキストでの「文化の変化」に対する、テスト自動化とCIの重要性を説明した資料です。
17. 17
Legacy Code: No test scripts!
Brownfield: Our Common Challenges
Test Automation
as a Technical Foundation
Improve design & program
Refactoring
Prevent regressions
18. 18
Lots of Immature Engineers
Engineers
Test Scripts
How to Write
Test Scripts
The Importance of
Test Automation
How to Test
Programs & Products
What should I do?
21. 21
Scope of Test Automation
Unit Testing
Acceptance Testing
Performance Testing
Security Testing
Other ility Testing
22. 22
Stages of Adapting to Unit Testing
1. Write simple test scripts
2. Adapt to TDD
3. Use Test Double (Mock)
4. Write characterization test if necessary
5. Refactor production code & architecture
23. 23
Schedule & Actions
Jan Feb Mar
Preparation
• Investigate products
• Create sample scripts
Execution
• Lectures
• Workshops
• Pair Programming
Execution
Remove architectural
impediments
24. 24
A powerful tool
to learn
Unit Testing & TDD!
Cyber Dojo: Our Weapon
http://cyber-dojo.org/
26. 26
Make product teams set up CI
processes
Our Goal
Nurture product teams
Avoid integration/deployment troubles
Achieve the target from CTO
Improve product teams
28. 28
e.g.) During integration & deployment process
Tons of Failures were There
Needed to avoid these troubles fundamentally
Staging
ENV
Production
ENV
CONFIG File
for Staging
Troubles!
31. 31
Accustomed to CI
Working
Samples
Team Members Coach
Worked with Samples
Configuration
of CI Server
CONFIG File
Did Pair Setting
(like Pair Programming)
Team Members Coach
32. 32
Always added Automated Tests
Test! Test! Test!
Pull the real power of CI
Adapt to Test Automation more
Nurture product teams
36. 36
Updated on Mar/31/2016
(during about 2 months)
Metrics
Item Quantity
Unit Tests Added/Updated 27
Software Bugs Detected 10
Infrastructure Bugs Detected 24
Prevented Troubles 3
It may be insufficient, but it’s REAL.
37. 37
Self-running Actions by Product Team Members
PO added “writing unit tests” as DoD
Started to write unit tests voluntarily
Established their own CI/CD strategy
Established their own testing rules
39. 39
Coaches should NOT do everything
Role of Coaches
For retaining learning opportunity
For nurturing voluntary actions
Giving hints is sometimes appropriate
Total
40. 40
Should Create Unavoidable Situation
CI:
Unavoidable from Test Automation
Top-down Approach:
Unavoidable to achieve objectives
Bottom-up Approach:
Unavoidable to scale improvements
Total
41. 41
Based on the production code
Sample Test Scripts Are Very Useful
Easy to adapt to Test Automation
Can use them as initial test scripts
Can learn their products & problems
Team
Members
Coaches
Test Automation
42. 42
Need to Solve Architectural Bottlenecks
Unable to apply mocks
Side effects
by static mocks
Hard to identify
web components
(Selenium WebDriver)
Modified interfaces
for injecting mocks
Introduced Phake
(Mock FW for PHP)
Defined locators
to each component
(like “id” or “name”)
Test Automation
43. 43
/**
* @test
*/
public function exampleOfTest() {
$sut = new Foo();
$result = $sut->bar('abc');
$this->assertEquals('huga', $result);
}
Pair Programming Clarifies Real SkillsTest Automation
48. 48
Made team members
learn a lot based on
automated working processes
Nurtured Product Teams
Fast Feedback
Learn from Failures
Lots of Confidence to Go ahead
CI
52. 52
Example of Additional Metrics
Test &
Test Automation
Code Coverage
Cyclomatic Complexity
# of Bugs Detected
# of Members Who Can Write Tests
Update Times of Test Scripts
(Lots of Updates = Valuable Test)
CI/CD Cycle Time
MTBF
MTTR
# of Releases per Week
# of Members
Who Can Operate CI/CD Servers
57. 57
Test Automation with CI
is sufficiently valuable
Adapting to Test Automation & CI in DevOps Context
Iterative & Incremental Improvement
Do Kaizen / 改善
58. 58
Adapting to Automation
is a long way
It’s Not Easy to Continue…
It takes 6 month
to do the same thing
for 1 team
in Facebook
Photo by Improve It - Kent Beck no Workshop Mapping XP.(2006) / CC BY-SA 2.0
Good morning, everyone! * 3
(Please respond loudly!)
OK, you are great, thank you
In this session, I will talk about DevOps from cultural aspect based on my experience.
Are you ready!?
OK, at first, please introduce myself.My name is Hiroyuki Ito.
So, please call me “The Hiro”. (The Greatest One in Japanese Software Engineering industry )
I belong to Yahoo Japan Corporation as an Automation & Agile Coach.
And, as you know, I have several certifications of Scrum.
Moreover, I made a presentation at Agile2014 in the U.S. (about 2 years ago)
Today, I will talk about the essence of this conference later.
Before starting the session, please give me 1 more minute to introduce my company.
In the session, please remember these 3 points at least
We are the technology-oriented company.
OK, it’s time to talk about DevOps.By the way, what is DevOps?(Questions to attendees * 3)
As you may know, there are lots of definitions of DevOps.
Here is one of famous definition, CALMS, or CLAMS.
But, here is my definition of DevOps.
My DevOps consists of these 3 parts.
2nd one (whole process improvement) means including management, design, development, test, operation and so on.
3rd one (technical foundation) means automation techniques like Continuous Integration and Continuous Delivery.
BTW, this is the famous chart from the book titled “Continuous Delivery”.
Quick show of hands. Who read this book?
My scope is those red ones.
We coaches are leading the cultural change especially based on Test Automation and Continuous Integration.
(Of course we use CD, cloud infrastructure and so on.)
We can efficiently shake up the organizational culture with Automated Test and CI.
And, I would like to explain this activity in the front at Yahoo Japan Corporation as the Experience Report.
And this is the prerequisite of our activities.
Please get some knowledge from my session (report).
Here is the agenda.
I will talk about Test Automation, CI, and evaluation of them.
Through these topics, I will explain the importance of cultural change in DevOps.
Let’s move on to Test Automation.
Here is our goal of Test Automation.
I think it’s a common sense.
To nurture self-running & cross-functional culture with higher quality & productivity.
Let’s check the background.
Here is the prerequisite of our products and teams.
We use PHP as main programming language.
Cloud-based infrastructure is our own OpenStack-based infrastructure like AWS.
It also means that OPS engineers provision with packages beforehand.
And, it caused lots of troubles later.
Do you know the meaning of “Brownfield”?
Brownfield means Legacy Code. And Legacy Code means no test scripts!
Brownfield problem is our common challenges in Yahoo Japan Corporation.
Yahoo Japan Corporation has a lot of long-lived services. For example, over 10 ages are common.
Therefore, we needed to implement Test Automation at first.
Additionally, we needed to solve the immaturity problem.
Half of engineers in product teams didn’t know these topics.
Can you imagine?
Moreover, we needed to overcome this passive culture with silos.
I know you love breaking silos
So we took the following challenges to get over these challenges.
There are a lot of Test Automation.We especially focused on Unit Testing because it’s easy to implement and effective to improve product development.
It’s the curriculum for immature engineers.
I think it is common sense for all of you.
Adapt to TDD : especially Test-First approach
We should include Refactoring with Test Automation!
And here is our schedule with actions.
At that time, we had only 3 months.
We executed these actions in Scrum manner.
We always improved our actions iteratively and incrementally.
Who knows Cyber Dojo?
OK, it’s a very powerful tool to learn Unit Testing and TDD.
I would like to demonstrate it
Did you enjoy Cyber Dojo?
OK, let’s move on to the next chapter: CI.
Are you READY!?
Here is our goal of CI.
We coaches supported them the same as Test Automation.
Let’s check the background.
To be honest, there were tons of failures like this.
Because of manual operations or complicated automated processes, they often happened.
At that time, we found a god
We can combine the top-down action by CTO with our bottom-up action.
Mike Cohn, a famous Agile Coach and one of the founders of the Scrum Alliance, also said the same thing is very important in his book titled “Succeeding with Agile”.
Here are our concrete actions for CI.
We made team members accustomed to CI with working samples and pair setting.
And, we always added automated tests.
We also simplified the complicated processes like this.
As you know, the right one is easier to understand than the left one.
Next, let’s evaluate our measures.
At first, let’s check the results.
Here is the metrics measured at the end of this March.
This is the result of our real actions during only 2 months.
Infrastructure bugs include the leakage of packages.
I think some people think it’s sufficient, but others are not.
But we detected some bugs as I wrote.
We went ahead a little. But it’s a real improvement.
These actions nurtured the self-running culture like this.
So I can say Test Automation & CI could shake up the culture.
Next, lessons learned throughout these activities.
First thing is about the role of coaches.
Coaches should not do everything.
Because extracting the power and ability from teams is very important.
Over support kills the power and ability from teams.
We would be better enforcing improvements by using these activities.
For example:
1) CI with Test Automation.
2) Top-down Approach for achieving objectives.
3) Bottom-up Approach for scaling improvements.
Unavoidable situation is a very powerful solutions for proceeding total improvements.
Next is the lessons learned from Test Automation.
We wrote and provided a lot of sample test scripts based on the production code.
Coaches should solve architectural barriers to Test Automation.
It's insufficient simply to make team members write tests.
Paying Technical Debts is also necessary.
I may show you very shocking scene.Don’t try this at your company
We knew that we need to reeducate them again and again…
But don’t get depressed!
We should say “That’s information, thank you!”
Because we can know the real problems in the teams and members.
Next is the lessons learned from CI.
We detected failures up-front with Test Automation like this.
Unexpected environmental changes means like EOL of DB servers.
Do you know this scientific truth?
Only PHP leads to Package Hell!
If you are using PHP, let’s introduce Test Automation and CI right now!
I strongly recommend it
We made them learn from failures based on trials and errors.
In our teams, CI is a driver for learning.
Test Automation, CI and CD can strengthen members, teams, organization, and products!
This is what I presented at Agile2014 as “Technology-Driven Development”.
It’s the additional possibilities of automation techniques!
Based on the above topics, we are planning to do these actions next.
Of course, we will proceed Continuous Delivery.
So I would like to share another themes.
Acceptance Test Automation : I especially love BDD
They are just information. You don’t need to remember all of them.
We can find a lot of metrics to distinguish the growth and health with them.
Let’s move back on to the starting point of this session.
What is DevOps?
In Yahoo Japan Corporation, we are challenging the cultural change as I talked above.
Test Automation with CI is sufficiently valuable in DevOps context.