Large, Multisite, & 'Offshore' Product Delivery with Large-Scale Scrum

1. Large, Multisite, &
‘Offshore’ Product Delivery
with
Large-Scale Scrum
v. 7
Craig Larman
Please...
Do not copy or share this material, or re-use for
other education.
Exceptions require prior written consent of the
author.
Copyright (c) 2013, Craig Larman. All rights reserved.
2 Craig Larman

2. scaling background...

3. serve as chief scientist @ Valtech
helped create “agile offshore”
lived in Bengaluru
5
was lead coach of lean development
6

4. various clients Scaling Lean & Agile
Development
started to experiment Thinking and Organizational Tools
for Large-Scale Scrum
with the scaling models Craig Larman
Bas Vodde
& tips from our
consulting
& books
curitiba, brazil helsinki, finland wroclaw, poland
for years, worked consulting/coaching worldwide at
large multi-site clients adopting large-scale Scrum
bengaluru, india antwerp, belgium hangzhou, china

5. my experience: single-product size?
150-2000 people
5-20 sites
5-100 MSLOC
median?
800? 5 sites? 10 MSLOC?
got tired of answering the same
questions, so wrote
(with Bas Vodde) ...

6. www.craiglarman.com
focus on (1) large-scale telecomm systems, and (2) large-scale financial systems ...
Scaling Lean & Agile
Development
Thinking and Organizational Tools
for Large-Scale Scrum
Craig Larman
Bas Vodde
11
Scaling Lean & Agile
Development
Thinking and Organizational Tools
for Large-Scale Scrum
Craig Larman
Bas Vodde
12

7. 13
first, a caution...

8. Scaling Lean & Agile
Development
Thinking and Organizational Tools
for Large-Scale Scrum
Craig Larman
Bas Vodde
16

9. organizational
design...
groups
roles
responsibilities
decision-making
control
life-cycle
rewards
hierarchy
planning
17
when scaling, the agile
values & Scrum concepts
sometimes get lost, because
the organization is so
“fixed” at the large scale...
18

10. organization...
teams
19
Scaling Lean & Agile
Development
Thinking and Organizational Tools
for Large-Scale Scrum
Craig Larman
Bas Vodde
20

11. “From interviews ... from the
“This new emphasis on speed
and flexibility calls for a different
CEO to young engineers ...
approach [iterative, overlapping we learned that leading
development]... The traditional companies show six
sequential approach to product characteristics in managing
development may conflict with their new product
the goals of maximum speed development processes:
and flexibility. ...
1.built-in instability
The [new approach] also does 2.self-organizing teams
away with the traditional notions
3.overlapping development
of division of labor...shared
division of labor, where each 4.multilearning
team member feels responsible 5.subtle control
for--and is able to work on--any
6.organizational transfer of
aspect of the system.”
learning”
21
“From interviews ... from the
“This new emphasis on speed
Sc
and flexibility calls for a different
CEO to young engineers ...
approach [iterative, overlapping we learned that leading
development]... The traditional companies show six
ru
sequential approach to product characteristics in managing
development may conflict with their new product
the goals of maximum speed development processes:
m
and flexibility. ...
1.built-in instability
The [new approach] also does 2.self-organizing teams
away with the traditional notions
3.overlapping development
of division of labor...shared
division of labor, where each 4.multilearning
team member feels responsible 5.subtle control
for--and is able to work on--any
6.organizational transfer of
aspect of the system.”
learning”
22

12. 23
specification product mgmt
group group
sys eng
program mgmt
group
group
component-1
programmers
firmware probably the
programmers
organizational
verification
structure before
group
adopting Scrum
doc group
24

13. specification product mgmt
group group
sys eng
program mgmt
group
group
component-1
programmers Scrum is NOT
firmware
for this group
programmers
verification
group
Scrum is not a
practice
doc group
25
specification Step 1: The separate
group groups are dissolved, and
sys eng members permanently join a
group long-lived Scrum Team;
component-1 there is still the mindset of
programmers “my single speciality”.
firmware
programmers
verification
group
doc group
26

14. specification Step 2: “team members” do
group multilearning & work on
sys eng tasks in secondary/tertiary
group
speciality; more pair-work.
component-1
Org focus on learning &
programmers
increasing flexibility.
firmware team
programmers Dyslexic
Zombies
verification
group
doc group
27
specification product mgmt
group group
sys eng
program mgmt
group
group
component-1
programmers
firmware
programmers
verification
what about
group
these groups?
doc group
28

15. organization...
the contract game
29
30

16. 31
this refers to the internal
contract created between
Product Management and R&D,
not external customer contracts
32

17. customer
external contract - OK
Product internal contract...
Management then what happens?
and/or
Account
Management
R&D
program managers
33
start end R&D
Product
(release)
Management
34

18. The Milestone point is arbitrary
start content freeze end R&D
Product
(release contract agreed) (release)
Management
The Contract
35
The Milestone point is arbitrary
start content freeze end R&D
Product
(release contract agreed) (release)
Management
The Contract
from Account or Product
Management perspective:
•low control
•low flexibility
•low transparency
•big batches
•can’t release early
36
•not “done” until “end”

19. * Development Phase for The Contract is controlled by R&D.
* The order of work is decided by R&D.
* Product Management does not have control, and there is low
visibility into the status of true progress.
start content freeze end R&D
Product
(release contract agreed) (release)
Management
ineffective bonus schemes and "tracking
to plan" behaviors are injected, since
The Contract there is no real control or visibility
37
more, 1
more,
more!
The Milestone point
is arbitrary
start content freeze end R&D
Product
(release contract agreed) (release)
Management
The Contract
38

20. more, 1 2 less,
more, less,
more! less!
The Milestone point
is arbitrary
start content freeze end R&D
Product
(release contract agreed) (release)
Management
The Contract
39
both parties in this 2-party
competitive game are
optimizing to shift blame
when there is a problem
40

21. The Milestone point is arbitrary
start content freeze end R&D
Product
(release contract agreed) (release)
Management
The Contract getting near
the “end”...
41
what happens in R&D in
response to “pressure”,
“incentives”, and “penalties” to
“get it all done”? (because R&D
made a “commitment” to “meet
the internal contract”)
42

22. 43
technical debt
learning & improvement debt
44

23. customer external contract - OK
Product pressure
Management
R&D
managers code
pressure learning
value
workers improvement
(developers, ...)
45
the subtle effects of an “internal contract”...
• batch delay
• the waste of WIP or inventory through late descoping
• speculative features (“over-production”)
• quality-destroying shortcuts
• reduction in transparency
• reduction in morale
• increased attrition (especially of best), ...
• learning, org, & technical debts, leading to slower &
slower development over time
46

24. your fault your fault
your fault your fault
start end R&D
Product
(release)
Management
The Contract
47
48

25. 49
Daily
Scrum
(15 min)
1 day
Sprint Sprint Sprint
Planning Planning 2-4 week Sprint Retro-
Part 1 Part 2 Sprint Review spective
(2-4 h) (2-4 h) (2-4 h) (1.5-3h)
Scrum
Feature
Team
Product + Product Backlog
Refinement Potentially
Owner ScrumMaster
(5-10% of Sprint) Shippable
Product
Increment
Product Sprint
Backlog Backlog
50

26. The Scrum Product Owner is from the business
area (e.g., head of Product Management or
Business Line), they are the investor.
The Product Owner has the responsibility for the
external contract, so they have the steering
wheel to drive (choosing/changing content and
date each Sprint)
51
customer external contract - OK
Product
Scrum team of
Manager
value workers
& Scrum PO
(developers, ...)
52

27. Scrum Roles
- Defines features of product, decides on release date & content
- Is responsible for the profitability of the product (ROI)
- Prioritizes features according to market value
- Can change features and priority every Sprint Product
- Accepts or rejects work results Owner
- Ensures that the team is fully functional & productive
- Enables close cooperation across all roles and
functions and removes barriers
- Shields the team from external interferences
ScrumMaster - Ensures that the process is understood (followed?)
- Cross-functional, seven plus/minus two members
- Has the right to do everything within the boundaries
of product groups guidelines to reach Sprint goal
- Organizes and manages itself and its work
53
- Reviews work results with the Product Owner Team
The PO is NOT a program manager,
system engineer, or other intermediate
R&D representative.
“fake PO” is not a PO!
The PO-investor steers directly.
54

28. The end of the “contract game” and
R&D “manages the release”
55
56

29. organization...
Product Owner
57
Product Owner is
owner of the Product --
responsible for ROI, etc.
58

30. 59
Scrum Roles
- Defines features of product, decides on release date & content
- Is responsible for the profitability of the product (ROI)
- Prioritizes features according to market value
- Can change features and priority every Sprint Product
- Accepts or rejects work results Owner
- Ensures that the team is fully functional & productive
- Enables close cooperation across all roles and
functions and removes barriers
- Shields the team from external interferences
ScrumMaster - Ensures that the process is understood (followed?)
- Cross-functional, seven plus/minus two members
- Has the right to do everything within the boundaries
of product groups guidelines to reach Sprint goal
- Organizes and manages itself and its work
60
- Reviews work results with the Product Owner Team

31. and agile (Scrum, ...) teams
are self-organizing (no project
manager directing teams),
so ...
61
before Scrum
(intermediate project/program managers)
analysis
architecture
R&D
product
project/ client-side
management
program
server-side
managers
test/QA
doc group
62

32. after Scrum
(Product Owner is 1 business owner)
team Beer
Product
Owner
1 Product team Dyslexic
Owner Zombies
from product
management
team
Dublin
63
Potentially Shippable
Product Increment &
Definition of Done
64

33. Potentially Shippable Product Increment
1 2 3 4 5 6 7 8 9 10 11 12
Potentially shippable
in practice...
Release DONE each Sprint
“Minimal Viable
Can release to customers at Product” takes several
the end of each Sprint, Sprints
though may not
65 Craig Larman
common misunderstanding:
“we must have customer-valuable
MVP increment each Sprint”
no! ... that would be nice, but is not
always possible (especially at scale)
66

34. we should be “PSPI”
every Sprint, but that
does not necessarily
mean “MVP” each Sprint
67
Potentially Shippable Product Increment
1 2 3 4 5 6 7 8 9 10 11 12
Potentially shippable
in practice...
Release DONE each Sprint
“Minimal Viable
Can release to customers at Product” takes several
the end of each Sprint, Sprints
though may not
68 Craig Larman

35. note that if PSPI each
Sprint... the definition of
MVP becomes flexible!
-> Business Agility
69
What is needed to be
potentially shippable?
70

36. Definition of Done
71
72

37. ...
73
what if the Sprint
Definition of Done is less
than “Release Done”?
74

38. Improving ‘done’ over time (lowering WIP)
2 year
improvement
10 year goal
goal
goal:
expand
customer marketing pricing
analysis doc
implement material
unit test customer update
performance production manufacturing
test test process
done
undone
needed to be potentially
current Definition of Done
shippable
75 Craig Larman
76

39. Potentially Shippable Product
Increment (or something
closer to it) each Sprint has
deep implications for business
flexibility ...
77
early business value
78

40. flexible decision about
when/what to deploy
79
flexible re-prioritization
each short cycle,
based on changes in
business conditions,
learning, etc
80

41. early feedback
81
early full-cycle
feedback
(documents don’t crash)
82

42. increased quality and
fitness for purpose
83
improved estimates
84

43. increased real visibility
85
fail fast
86

44. integrate early
87
reduced WIP
88

45. reduced batch delay
89
Release Sprint
90

46. what if the “definition of
done” is not perfect?
91
Improving ‘done’ over time (lowering WIP)
2 year
improvement
10 year goal
goal
goal:
expand
customer marketing pricing
analysis doc
implement material
unit test customer update
performance production manufacturing
test test process
done
undone
needed to be potentially
current Definition of Done
shippable
92 Craig Larman

47. Handling Undone Work - by Team
(undesirable, but sometimes inevitable)
www.craiglarman.com
www.odd-e.com Product Owner Release Sprint:
Copyright © 2010 wants to release teams only do
C.Larman & B. Vodde
All rights reserved. Undone Work
...
Undone
Work actual
release
93
Handling Undone Work
(undesirable, but sometimes inevitable)
Release Sprint:
teams +
Undone Unit teams start work
Product Owner on next release
do pair-work
wants to release
on the
Undone Work
...
Undone interruptions for
Work handoff
unantipicated problems
www.craiglarman.com
www.odd-e.com completing the actual
Copyright © 2010 Undone Undone Work release
C.Larman & B. Vodde
All rights reserved. Unit
94

48. eventually, eliminate a
Release Sprint (and the
Undone Unit) by
perfecting “done”
95
if a Release Sprint is still
necessary, consider the
“Rule the 3” for “semi
Release Sprints”
96

49. Thinking about...
programmers
97
98

50. do you have a “resource” mentality,
where developers are “resources” or
“heads” and, as in manufacturing, we
add “more resources to produce
more doughnuts”?
99
“let’s outsource to South Yemen
because programmers are cheaper
there... and programmers only have
90% variation in productivity”
really? ...
100

51. 101
Large meta-analysis shows the top quartile
of developers is at least 4 times faster than
the bottom quartile
– [Prechelt00]
102

52. “The difference between the best worker on
computer hardware and the average may be
2 to 1, if you’re lucky. With automobiles,
maybe 2 to 1. But in software, it’s at least 25
to 1. The difference between the average
programmer and a great one is at least that.”
– Steve Jobs
104

53. organization...
component
teams?
105
Scaling Lean & Agile
Development
Thinking and Organizational Tools
for Large-Scale Scrum
Craig Larman
Bas Vodde
106

54. component teams
Programmers specializing in one component/module/layer/
application
GUI team, platform team, device driver team, component-1
team, component-2 team, ...
Comp-1 Comp-2 Comp-3
Comp-4 Comp-5 Comp-6
107 Craig Larman
the problem? ...
108

55. Analysis
Backlog Item 1 Design
Backlog Item 2
Backlog Item 3
Backlog Item 4 realistically, not Implementation
... available as
soon as the realistically, not all
analyst is teams start on Item 1 Test
Item 1 finished programming at the
same iteration; they are
multitasking on many it is unlikely that the
partially done features system testers are
available to test
Analyst System
Comp A Item 1 as soon as
Engineer the last component
Team
team has finished
code
Comp B
requirement Team
details
for Item 1
System
Comp C Testers
tasks by Team
component 109
Analysis
Backlog Item 1 Design
Backlog Item 2
Backlog Item 3
Backlog Item 4 realistically, not Implementation
... available as
soon as the realistically, not all
analyst is teams start on Item 1 Test
Item 1 finished programming at the
same iteration; they are
multitasking on many it is unlikely that the
partially done features system testers are
available to test
Analyst System
Comp A Item 1 as soon as
Engineer the last component
Team
team has finished
code
Comp B
requirement Team
details
for Item 1
System
Comp C Testers
tasks by Team
component 110

56. Analysis
Backlog Item 1 Design
Backlog Item 2
Backlog Item 3
Backlog Item 4 realistically, not Implementation
... available as
soon as the realistically, not all
analyst is teams start on Item 1 Test
Item 1 finished programming at the
same iteration; they are
multitasking on many it is unlikely that the
partially done features system testers are
available to test
Analyst System
Comp A Item 1 as soon as
Engineer the last component
Team
team has finished
code
Comp B
requirement Team
details
for Item 1
System
Comp C Testers
tasks by Team
component 111
Analysis
Backlog Item 1 Design
Backlog Item 2
Backlog Item 3
Backlog Item 4 realistically, not Implementation
... available as
soon as the realistically, not all
analyst is teams start on Item 1 Test
Item 1 finished programming at the
same iteration; they are
multitasking on many it is unlikely that the
partially done features system testers are
available to test
Analyst System
Comp A Item 1 as soon as
Engineer the last component
Team
team has finished
code
Comp B
requirement Team
details
for Item 1
System
Comp C Testers
tasks by Team
component 112

57. note how organizational
structure inhibits change in
architectural structure -- and
vice versa
(Conway’s observation)
113
component teams
Programmers specializing in one component/module/layer/
application
GUI team, platform team, device driver team, component-1
team, component-2 team, ...
Comp-1 Comp-2 Comp-3
Comp-4 Comp-5 Comp-6
114 Craig Larman

58. our code, subsystem, custom
tool, framework (not yours)
internal open source
organization...
large-scale Scrum
framework 1
116

59. how can that 1 PO be the
PO for 5 Scrum Teams?
(i.e., interact effectively
with 5 teams)
117
118

60. one Sprint per Product, not per Team
Sprint N Sprint N+1
119
one Product Backlog per Product,
not per Team
---
---
---
Product Product
Owner Backlog
120

61. Daily
Scrum
Scrum (15 min)
Feature
Team
+ 1 day
ScrumMaster
2-4 week
Sprint Sprint
Planning
Part 2
(2-4 h)
Sprint
Sprint Sprint Product Backlog Retrospective
Planning Backlog Refinement (1.5-3h)
(5-10% of Sprint) Sprint Joint
Part 1
(2-4 h) Review Retro-
(2-4 h) spective
Potentially
Product Shippable
Owner Product
Increment
Product
Backlog
large-scale Scrum
up to 5 or 10 teams 121
framework 1
122

62. organization...
large-scale Scrum
framework 2
123
requirement areas...
124

63. Scaling Lean & Agile
Development
Thinking and Organizational Tools
for Large-Scale Scrum
Craig Larman
Bas Vodde
125
Product Backlog
Backlog Item Requirement Area
IPv6 protocols
performance 10x performance
HSDPA management
performance stats protocols
configuration of cells management
new NMS solution continuous integration
speed-up of build upgrades
improved upgrading support management
stability to 99.999% reliability
126

64. Performance Area Backlog
Product Backlog
performance 10x switch hardware
IPv6 performance 10x optimize DSP
performance 10x ... ...
HSDPA
performance stats
configuration of cells
new NMS solution Protocols Area Backlog
speed-up of build
improved upgrading support IPv6 simple connect
stability to 99.999% IPv6 data sending
HSDPA failed call
... ...
127
performance area feature teams
or, “overall PO” Area
feature feature feature
Product
team team team
Owner
Performance
Product Backlog
Backlog Items 1 feature feature feature
Backlog Item 1 Backlog Items 2 team team team
...
…
Protocols
... feature feature feature
Backlog Item 3 team team team
Backlog Item 4
...
feature feature
team team
Area
Product
Owner
protocols area feature teams
128

65. Daily
Scrum
Scrum (15 min)
Feature
Team
+ 1 day
ScrumMaster
2-4 week
Sprint Sprint
Planning
Part 2
(2-4 h)
Sprint
Sprint Sprint Product Backlog Retrospective
Planning Backlog Refinement (1.5-3h)
(5-10% of Sprint) Sprint Joint
Part 1
(2-4 h) Review Retro-
(2-4 h) spective
Potentially
Product Shippable
Owner Product
Increment
Product
Backlog
large-scale Scrum
up to 5 or 10 teams 129
framework 1
Daily Scrum
Scrum Feature (15 min)
Team
+
ScrumMaster 1 day
Sprint
Planning 2-4 week
Part 2 Sprint
(2-4 h)
Sprint Product Backlog
Sprint Backlog
Planning Refinement
(5-10% of Sprint)
Part 1
(2-4 h)
Joint Retrospective
Potentially
Sprint Review
Product Area Shippable
Owner Product Product
Sprint Retrospective
Owner Increment
Product
Backlog Area
Product
Backlog
large-scale Scrum
works for hundreds of teams 130
framework 2

66. organization...
CoPs
131
132

67. CoPs for all cross-discipline concerns
133
organization...
design &
architecture
134

68. 135
vertical slices, not horizontal components/
frameworks/...
AKA: use-case driven, feature-driven,
tracer-bullet development, walking skeleton
136

69. vertical slices, not “parts”
iterative & evolutionary, not “build components”
example from: Jeff Patton
137
CoP for design/architecture

70. design workshops &
joint design workshops
agile architecture documentation workshop:
N+1 View Model, Videos, Technical Memos
Chapter 39:
Documenting
Architecture

71. agile architecture documentation workshop:
N+1 View Model, Videos, Technical Memos
“PowerPoint” architects
culture of master-programmers
culture of gardening, not “building”

72. system-engineering group
architecture group
...
branching for customization
configuration for customization
(metadata, ...)

73. our code, subsystem, custom
tool, framework (not yours)
internal open source
component stewards

74. organization...
managing ‘big’
requirements
147
148

75. smaller-requirement splitting (not by tasks)...
use cases configurations
I/O channels (e.g., via
CRUD use cases
GUI and non-GUI)
scenarios of a use case
data formats
data parts
role or persona
“type” specialization
“non-functional”
(e.g., different types of requirements
trades)
operation (e.g., HTTP
scenario steps GET, PUT)
integration with external with stubs
elements 149
DHCP server support
request an IP address // use-case
renew an IP address // use-case
all other use cases // lazy splitting
150

76. Request an IP address
... when any free one is available // scenario
... when a specific one is requested and it is free // scenario
... when none are free, fail & notify // scenario
... all other scenarios // lazy splitting
151
HSDPA calls
... when any connection failure, fail & notify
... all other scenarios // lazy
152

77. organization...
multisite
153
154

78. terminology:
dispersed team
(avoid them; they don’t scale
for 500-person groups)
155
large-scale multisite
development does not require
dispersed teams
156

79. Sprint N Sprint N+1
Scrum Scrum
Feature Feature
1 Sprint per
Team Team
London
Scrum Scrum
Feature
Team
Feature
Team product, not
per site
Scrum Scrum
Feature Feature
Shanghai Team Team
Scrum Scrum
Feature Feature
Team Team
157
whole feature to co-located non-
dispersed Scrum Team
Shanghai Team Blue
Scrum Team
long-lived, cross-functional
potentially
customer- shippable
Subject product
centric Developer Customer Doc
Expert increment
feature
Product
Owner
Tester Interaction
Architect
Designer
Analyst
158
This figure could be misinterpreted: A Scrum Team does not have a
person who is only a Developer and does not have a person who is
only a Tester. Rather, people have primary skills such as Developer and
Tester, and also other skills—and are learning new areas. Team

80. performance area feature teams
Area
feature feature feature
Product
team team team
Owner
Shanghai
Performance
Product Backlog
Backlog Items 1 feature feature feature
Backlog Item 1 Backlog Items 2 team team team
...
…
Protocols
... feature feature feature
Backlog Item 3 team team team
Backlog Item 4
...
feature feature
team team
Area
Product
Owner
protocols area feature teams
159
sites are equal partners
160

81. continuous integration, across all sites worldwide
161
162

82. seeing is believing...
ubiquitous free video technology in team rooms
163
multisite matchmakers, not
intermediaries
164

83. multisite communications CoP
165
site-level Joint Retrospective
166

84. commercial tools
167
documents (Word, Excel
1985
SharePoint...)
168

85. organization...
inspect & adapt
169
170

86. Sprint N Sprint N+1
Team-level Sprint Review
Joint Review Joint Retrospective
Team-level Sprint Retrospectives
171
Impediments Backlog;
Impediments
Service
Transformation
Backlog;
Transformation
Project
172

87. closing
173
these are large and context-
sensitive topics, best
understood in the upcoming
discussions or via the books...
174

88. www.craiglarman.com
Scaling Lean & Agile
Development
Thinking and Organizational Tools
for Large-Scale Scrum
Craig Larman
Bas Vodde
175