This document discusses process and contracting innovations that are key success factors for government IT projects. It summarizes three case study projects in Norway that utilized agile methodologies. The largest project was completed successfully, while the two ongoing projects faced challenges. The document recommends establishing a high-quality product backlog, integrating testing into development sprints, using a release master plan and control gates for planning and monitoring, and implementing a balanced target price contract with 50/50 risk sharing between the customer and supplier.

1. Process innovation - Key Success Factor
in Government Projects
Remi Hansen, PROMIS AS
Kristina L. Tangen, EVRY ASA
25.09.2014 • © PROMIS AS 1

2. 25.09.2014 • © PROMIS AS 2
Agenda
1. Introduction to the presenters and the case projects
2. Process innovations
• Backlog and business value innovations
• Test and approval innovations
• Planning and monitoring innovations
3. Contracting innovations
• Agile contracting - balanced contracts
• Target pricing, incentives and risk sharing
4. Q & A

3. The Presenter – Kristina Lassen Tangen
• Computer Science from University of Oslo
• Certified Scrum Master
• Certified ISEB Practitioner
• More than 20 years experience from IT-business
• System developer
• Project and test Manager
• Head of test manager community
• Currently Chief Consultant in EVRY and head of test management community
• Experienced in Agile project – from 2003
25.09.2014 • © PROMIS AS 3

4. The Presenter – Remi Hansen
• B.Sc. In SW Engineering, M. Sc. in Industrial Economics
• More than 20 years experience from
the IT Consulting business
• System Developer (3 years)
• Project Manager and Business Consultant (14 years)
• Line manager (4 years)
• Certified Project Management Professional (PMP), PRINCE2 Practitioner,
IT Project Professional (ITPP), CSPO, ISTQB SW Testing Foundation and ITIL.
• Experience with agile projects started in 2002.
• Currently Senior PM in PROMIS, a leading provider of agile project
management services in Norway (www.promis.no)
25.09.2014 • © PROMIS AS 4

5. Some pricing model basics
25.09.2014 • © PROMIS AS 5

6. Background understanding
Contracting price models
• Time & Material:
• The Customer pays for all worked hours, disregarding productivity and
quality – all risk is on the Customer
25.09.2014 • © PROMIS AS 6
• Fixed Price:
• The Customer pays the same amount, disregarding the hours needed to
produce the scope with agreed quality – all risk is on the Supplier
• Target Price:
• Customer and Supplier share benefits or losses in an agreed ratio
– if this ratio is 50%, the risk is evenly distributed between the parties

7. The hourly rate curves –
comparing different price formats
25.09.2014 • © PROMIS AS 7

8. The case study projects
25.09.2014 • © PROMIS AS 8

9. 25.09.2014 • © PROMIS AS 9
Case study
• Lessons learned from 3 federal
government projects
• Over 1,200,000 hours of effort
• Not a scientific study
• 1st and 2nd hand information used
• Factual information is from public sources,
the rest is partly our personal opinions and
partly from sources listed
Photo (Flickr):
Okko Pyykkö

10. Norwegian Public Service Pension Fund (SPK)
PERFORM project
• The largest and most important project ever
for the Public Service Pension fund
• Government funded
• About 800 000 project hours spent
and 100-180 persons involved
• Implements system support for managing new
25.09.2014 • © PROMIS AS 10
pension reform
• Replacing legacy system due to outdated
technology
• Agile development methodology - Scrum
• Status: Completed – very successful, seen as
best practice in agile implementation

11. Statnett
LARM Project
Statnett
• State Enterprise responsible for all high
voltage electricity transmission
and distribution in Norway
• Owns, operates and regulates
the national main grid
• Statnett co-ordinates supply and demand, and owns the main Norwegian power grid.
25.09.2014 • © PROMIS AS 11
Photo: Statnett.no

12. Statnett
LARM Project
• Community-critical system
• Ensuring the responsiveness of the national power grid
• Large project
• Interesting feature
• Very specialized domain – few users, but community critical operation
• Contract and sourcing setup
• Single supplier
• Target price with customer friendly profile
25.09.2014 • © PROMIS AS 12
• Status:
• Ongoing (midway)
• Challenged – overruns and conflicting interests
Photo: Statnett.no

13. The Norwegian Public Roads Administration
AUTOSYS project
The NPRA
• Responsible for the planning, construction and operation of the national and
county road networks, vehicle inspection and requirements, driver training and
licensing
25.09.2014 • © PROMIS AS 13
Autosys project
• Replacing 30 year old mainframe system
• 5 releases in 3 years (overall 5 year project)
• SOA platform
• Size:
• 65 on supplier side, 35 on customer side
• 4 Scrum teams
• € 100 million project budget
• Status: Ongoing, challenged Photo: Vegvesen.no

14. Case study projects summarized
• Large to very large projects in public
25.09.2014 • © PROMIS AS 14
sector
• Very different domains and parts of the
state administration
• All Scrum based projects
• One successfully completed,
two ongoing and challenged
• All rely on IT suppliers governed by
contractual obligations
• All target price contracts, but with very
different customer / supplier balance
Photo (Flickr):
jardenberg

15. 25.09.2014 • © PROMIS AS 15
Agenda
1. Introduction to the presenters and the case projects
2. Process innovations
• Backlog and business value innovations
• Test and approval innovations
• Planning and monitoring innovations
3. Contracting innovations
• Agile contracting - balanced contracts
• Target pricing, incentives and risk sharing
4. Q & A

16. Process innovation: Business value and
product backlog innovation
25.09.2014 • © PROMIS AS 16

17. Case findings
Business value and product backlog issues
• Poor quality of the product backlog is devastating for the project
• Decision making: real (and lasting!) prioritization is difficult for inexperienced
25.09.2014 • © PROMIS AS 17
P.O.s
• Responsibilities for different aspects of the product backlog is a source of
misunderstanding and controversy

18. Recommendations
Good product backlog practice
• The customer must control the scope!
• A well organized P.O. Team supporting the Chief P.O. is needed
• Individual P.O.s must coordinate across domains – creating one coherent
25.09.2014 • © PROMIS AS 18
backlog
• Appoint a Business Analyst to each sprint team = P.O.’s delegates to the
teams will relieve the load on the P.O.
Photo (Flickr):
Budzlife

19. Product owner team
Product owner team
Development team 1 Development team 2 Development team 3
25.09.2014 • © PROMIS AS 19
Product owner

20. Recommendations
Good product backlog practice
• Continuously work with grooming the backlog – invest enough capacity to
clearly define User stories, and give priority
• Setting priorities takes a lot more effort than one should expect
• Put in place a method for assessing business value
• View the User story estimate as the budget for the task – budgeted effort is set
in the contract  the scope must be controlled to fit that effort
25.09.2014 • © PROMIS AS 20

21. Business value and product backlog issues
Key Learning Points
• The quality of the backlog cannot be stressed enough –
the project success completely relies on it!
 Ensure enough effort is spent grooming the backlog
• Use business value as basis for prioritization
• Set up a P.O. team as described and appoint a
Business Analyst to each scrum team
25.09.2014 • © PROMIS AS 21
Photo (Flickr):
Horia Varlan

22. Process innovations –
Test and approval innovations
25.09.2014 • © PROMIS AS 22

23. System Hardening sprints
• 2-3 sprints after the development sprints to perform system test and complete
outstanding work (typically documentation and clean-up) – done in parallel with
solution description for the next delivery
• Is that a good idea?
 There’s always some activities not completed. May give better preparation for and
smoother execution of Acceptance test
 Real integration test will not be done along the way  becomes a waterfall
approach, where testing is postponed (long time from programming to testing)
 Risk mitigation is delayed
 Very demanding for the project to do both system testing and solution design in
parallel
 Probably not a good idea – include system testing in the development sprints
instead to minimize time from programming to testing
25.09.2014 • © PROMIS AS 23

24. Unit test
Integration test –
Continuous integration
Functional System test
and Integration test
Approval test
Acceptance test
Production test
25.09.2014 • © PROMIS AS 24
Test strategy
• Different system test approaches taken – integration of test into development
seems to give the best result
Development cycle
Responsibility:
Development teams

25. Test process - development
Iteration n - 1 Iteration n Iteration n + 1
Define and
execute
Functional
system tests
Define and
refine
functional
test
conditions
Define and
execute
unit and
integration
tests
25.09.2014 • © PROMIS AS 25
Acceptance
criteria
Execute System
Integration tests

26. Recommendations
Integration of test and development tasks
• Business Analysts develop acceptance criteria for each user story before
25.09.2014 • © PROMIS AS 26
coding
• TDD at all test levels
• The Test Manager should state quality requirements to development teams –
i.e. Definition of Done related to testing
• The Test Manager should have the authority to label user stories «Done»
• Run tests in fully integrated system
• There should be very limited need for the customer to perform separate tests –
rather verifying the test activities undertaken by the supplier
• Turn mindset of customer from traditional waterfall to agile

27. Recommendations
Integration of test and development tasks
• Integrate test into Scrum teams
• Appoint one in each Scrum team as
Team Test Responsible
• Planning, design, test conditions,
development and test performed
collectively by the team
• Testers are no longer quality police
– instead valued members of the team
• Unit tests, integration tests, functional
system test and system integration
tests in construction sprints
25.09.2014 • © PROMIS AS 27

28. Integration of test and development tasks in the sprint teams
Key Learning Points
 Maximize the testing done as part of the development
25.09.2014 • © PROMIS AS 29
sprints
 Appoint a test responsible in each sprint team
 Empower the Test Manager to set process
requirements
 Avoid pointless replication of tests on the Customer
side
Photo (Flickr):
Horia Varlan

29. Process innovations – Planning and
monitoring innovations
25.09.2014 • © PROMIS AS 30

30. Recommendations
Planning and monitoring
• A release master plan (product roadmap) – not just a product backlog
• Control gates: Book keeping for each sprint (earned value calculations etc) –
25.09.2014 • © PROMIS AS 31
covered later
• Target price on user stories – not on release. Makes it possible to reprioritize
and move user stories in / out of scope

31. Recommendations
Planning and monitoring – Release Master Plan
• The Release Master Plan is a tool for planning handover and implementation of
new functionality
• The Release Master Plan identifies the number of releases, their size, timing
and dependencies
• Guiding principle: Realization of business value!
25.09.2014 • © PROMIS AS 32

32. Recommendations
Planning and monitoring – Control gate at sprint completion
• Definition of done
• May be done in 2-3 working days following the sprint demo
• During these days:
• Functional verification by Product Owner and his / her team
• Checking code quality
• Checking architectural guidelines
• Checking test documentation
• Checking other documentation
• Contractual milestone
25.09.2014 • © PROMIS AS 33

33. The Anatomy of the Sprint
Sprint demo Sprint X
Sprint planning Sprint X+1 starts with decomposition
Sprint backlog for Sprint X+1 ready
Control gate:
1. Evaluation of Sprint X
2. Approval of sprint plan for Sprint X+1
3. Updating the risk matrix
4. Change control
5. Repatriation of agreed outstanding issues to the
product backlog
25.09.2014 • © PROMIS AS 34

34. Construction phase: The control gates
Construction phase
Blue line: Sprint team
Red line: Product Owner
Yellow line: Verification, Control gate and Definition of Done
25.09.2014 • © PROMIS AS 35
Solution description
phase

35. Planning and controlling considerations in CG
 What was the productivity of the last sprint?
 How was the product quality?
 How is the risk picture evolving?
 What can we improve – emphasis on learning and continuous improvements
25.09.2014 • © PROMIS AS 36

36. Planning and monitoring –
Key Learning Points
 Use a Release Master Plan for planning and
25.09.2014 • © PROMIS AS 37
communication
 Use Earned Value Management – based on user story
estimates – not effort on activities
 S-curve with planned value, earned value and actual
cost
 Calculating productivity (Cost Performance Index) in
Control Gate process as a basis for forcasting
 Execute a Control Gate process after each sprint for
control (progress / velocity, productivity, risk, etc.)
Photo (Flickr):
Horia Varlan

37. 25.09.2014 • © PROMIS AS 38
Agenda
1. Introduction to the presenters and the case projects
2. Process innovations
• Backlog and business value innovations
• Test and approval innovations
• Planning and monitoring innovations
3. Contracting innovations
• Agile contracting - balanced contracts
• Target pricing, incentives and risk sharing
4. Q & A

38. Part II: Contracting innovations
25.09.2014 • © PROMIS AS 39

39. Why contracts?
• Public sector is engineered for cost control – «must» have a contract with
supplier obligations to have cost limit
• Any customer will want supplier buy-in – achieved through risk sharing /
25.09.2014 • © PROMIS AS 40
incentives
Photo (Flickr):
Images_of_Money

40. What price model is beneficiary in agile projects?
Target price is best suited because:
• Agile means close co-operation
• 50/50 sharing of incentives and sanctions
ensure the customer and supplier works
towards the same goal
• Suitable when requirements are not very
25.09.2014 • © PROMIS AS 41
detailed
• Can be based on a less detailed basis than
traditional fixed price
Price model Average overrun
Time & Material based
(4 projects)
55%
Fixed price (5 projects) 33%
Target price (7 projects) 10%
Others (2 projects) 13%
Total (18 projects) 27%
• Fixed price assumes
requirements are described in
detail and that uncertainty is
low.
• T & M is suitable when scope is
not well defined and uncertainty
is high.

41. Why Target Price Contracts with Risk Sharing?
• The Supplier is invited to commit to estimates with a higher degree of
uncertainty than in traditional fixed price contracts
• Support agile principle to avoid waste
• Reduced effort can be on initial specification, detailed design and planning
• The Customer may produce tender documents with high level requirements
• The Suppliers produce solution descriptions, estimated with a significant degree of
25.09.2014 • © PROMIS AS 42
uncertainty
 Both parties reduce work load in the initial bidding phases of the project

42. Recommendations
Contracting
A balanced contract is necessary to ensure commitment and quality
• Implies 50 / 50 risk sharing with no cap
• Not agile to commit to up-front estimates for several years based on very
limited knowledge – risk for the supplier and ultimately the customer
• Include mechanisms to avoid commitment to estimates over a very long time
25.09.2014 • © PROMIS AS 43

43. PS2000 – a Target Price Contracting standard
• A Norwegian IT contracting standard, available in international English version
• Balanced contract, addressing issues seen in past projects (“best practice”)
• Iterative and agile version
• De facto standard for large agile projects in Norway
 There is a well-proven agile contract available
Progression
CG1
Iterative
construction phase
Solution
Description
http://www.dataforeningen.no/it-contract-standards.146223.no.html
25.09.2014 • © PROMIS AS 44
Requirement
Analysis
Acceptance an-d
Completion phase
Detailed planning
Analysis and design
Testing Development
CGn
C
CG2
PMS 0
Contract Signing
PMS 1
Approved Solution
Description
PMS 2
Delivery ready
for Acceptance
PMS 3
Accepted
Delivery

44. In summary - Advantages with PS2000 Agile
• Describes how the project phases should be executed
• Loyal to the principles in Agile – roles, ceremonies and artifacts from Scrum
• Target price model, with a large variety of configurations reflecting project risks
 Incentives for the supplier to deliver more functionality within agreed time
limits, but with satisfactory quality
• Fast and easy to complete annexes
• Suitable for repeating releases in a frame agreement for new development or
application maintenance
• The main success factor is that the parties agree on the process for managing
the Product Backlog, Sprints and Control Gates, with the corresponding roles
• Change control becomes lean – a signed Product Backlog after each sprint
documents the agreed changes
25.09.2014 • © PROMIS AS 45

45. Specific public sector challenges and contractual implications
Key Learning Points
 Agile projects can be run under a contract, given that
the contract
 Is balanced
 Is engineered for agile execution
 Lets scope be defined as late as possible
- preferably contracting each release as a separate
target price delivery
- ensuring estimates are as realistic as possible
 PS2000 Agile is such a contract standard
25.09.2014 • © PROMIS AS 46
Photo (Flickr):
Horia Varlan

46. 25.09.2014 • © PROMIS AS 47
Summary

47. Summarizing some major issues
• Impression that all the projects would be agile – only one of them were
• Contractual constraints
• Cultural challenges – a fixed price mindset on the customer side
• Scrum does not make a project agile!
• Naive to expect Scrum / agile to solve every problem
• Even the most successful project had challenges in the first stage – but a
willingness to learn & adapt brought the project on the path of good practice
25.09.2014 • © PROMIS AS 48

48. Some smart moves in the PERFORM project
paving the way for success
• Gained experience with Scrum before starting the project
• Understood the value of a balanced contract
• Initiation phase on T&M price model
• The customer had own Scrum teams
in parallel with supplier teams
• Invested in good procedures and tools for
migration and configuration control
• Test as integrated part of development
• Hired external help for agile project management
coupled with active top management involvement
• The customer took overall responsibility for all processes – the main suppliers
had responsibility for construction of their parts for every release
• A continuous focus on improvement
 Truly agile execution!
25.09.2014 • © PROMIS AS 49
Photo (Flickr):
apdk

49. 9 defining features of agile projects
1. Business value is the most important criteria for quality and direction
2. Continuous prioritization of functionality based on cost / benefit
3. Close communication between the business side and developers
4. Short iterations to delivery of executable code
5. Frequent releases to production (integrated and tested)
6. Binding decisions taken as late as possible (‘Rolling Wave Planning’)
7. Evaluation, learning and improvement along the way
8. Autonomy: Self-organizing cross-functional teams
9. Avoid waste
25.09.2014 • © PROMIS AS 50

50. Defining features of agile projects
Feature Degree of
divergence in case
study projects
25.09.2014 • © PROMIS AS 51
Covariance
between good agile
practice and
success
Business value focus High
Continuous prioritization High
Communication with business Medium-High
Short iteration – executable code Low
Frequent releases Low
Deferring decisions High
Learning Medium-High
Autonomy Low
Avoid waste High

51. Public sector specifics?
• Wide variety of organizations, management and culture in public sector
– not possible to see the whole sector as one homogeneous group
• The recommendations given here applies equally to private sector projects
25.09.2014 • © PROMIS AS 52

52. 25.09.2014 • © PROMIS AS 53
References
• IT Project Professional certification curriculum - http://smidigeprosjekter.no/itpp
International launch of certification in 2013.
• Upcoming book – working title ‘Agile Contracting’
• PS2000 Agile Standard Contract
• http://www.dataforeningen.no/it-contract-standards.146223.no.html
• Complete English version - PS2000 Agile (PS2000 Agile, Maintenance, Framework,
Service Operations) ~ € 750
• PRINCE2

53. Questions or comments?
Photo (Flickr):
Horia Varlan
25.09.2014 • © PROMIS AS 54

54. Thank you for the attention!
rh@promis.no
kristina.tangen@evry.com
25.09.2014 • © PROMIS AS 55