Introduction to the International Software Benchmarking Standards Group and 3 cases in which function points together with ISBSG data really resulted in business value:
- Reality check of an estimate made by experts
- Assessing the competitive position of a department
- Selecting a single software supplier
2. Harold van Heeringen
President ISBSG
International Software Benchmarking Standards Group (www.isbsg.org)
Board member NESMA
Netherlands Software Metrics Association (www.nesma.nl)
IAC member COSMIC
Common Software Measurement International Consortium (www.cosmicon.com)
Senior Cost Engineer,
Sogeti Nederland B.V. (www.sogeti.com)
@haroldveendam
harold.van.heeringen@sogeti.nl
3. Overview
ISBSG introduction
IFPUG and ISBSG in practice: 3 cases
Reality check of an expert estimate;
Competitiveness assessment of an IT department;
Selecting the right (sole) supplier to do all the IT and
manage the supplier based on productivity.
ISBSG: other useful purposes
How does Spain compare to the industry
4. We measured 350 FP, so... What is the estimate?
Effort, duration, quality, risk?
Price per FP?
Why are we losing most of the bids we do?
Competitive advantage? Or not?
What is our productivity compared to the industry?
We are going to send out a request for bid based on price/FP
How can we compare the proposals?
How can we exclude unrealistic (overoptimistic) ones?
What should the market average bid approximately look like?
We did a project and the client thinks it was too expensive
How did we perform against similar projects?
Size does matter, but….
5. Historical data of completed projects!!
The answer…
Without software metrics based on an objective
unit of measure, there is little we can do. To be
able to analyze and to make decisions, we need:
6. International Software Benchmarking Standards Group
Independent and not-for-profit;
Full Members are non-profit organizations, like IFPUG, NESMA,
GUFPI-ISMA, FiSMA, QESP, DASMA, JFPUG, Swiss-ICT and CESI;
Associate members: AEMES (Asociacion Espanola de Metricas de
Software), ASSEMI (France);
Grows and exploits two repositories of software data:
New development projects and enhancements (> 6000 projects);
Maintenance and support (> 1200 applications).
Everybody can submit project data
DCQ on the site / on request (.xls)
Anonymous
Free benchmark report in return
ISBSG
7. Mission: “To improve the management of IT resources by
both business and government, through the provision and
exploitation of public repositories of software engineering
knowledge that are standardized, verified, recent and
representative of current technologies”.
All ISBSG data is
validated and rated in accordance with its quality guidelines
current
representative of the industry
independent and trusted
captured from a range of organization sizes and industries
ISBSG
9. ISBSG wishes to help restore the confidence in IT, by
giving transparency of what is realistic, resulting in
Improved project success rates;
Better customer satisfaction;
Shorter time to market;
More efficient software development;
Better alignment with the business/customer;
Easier contracting, resulting in better relationships
between customer and supplier.
ISBSG IT Confidence
10. 3 Cases from my experience
Case 1: Telecom project reality check on the expert
estimate (project manager)
Case 2: Assessing the competitive position of an
organization (Senior management)
Case 3: Supplier Performance Measurement
(procurement)
12. Case 1
A telecom company wished to develop a new Java system for the
maintenance of subscription types;
A team of experts studied the requirements documents and
filled in the WBS-based estimation calculation (bottom-up
estimate);
They decide that an estimate of 5.500 hours and a duration of 6
months should be feasible;
The project manager decided not to believe the experts ‘on their
blue eyes’ only and wished to carry out a reality check.
13. Why a realistic estimate?
Non-linear extra costs
-Planning errors
-team enlargement more expensive, not faster
-Extra management attention / overhead
-Stress: More defects, lower maintainability !!
Linear extra costs
Extra hours will be used
14. ISBSG Reality Check: Effort
An estimated FPA comes up with the expected size:
Min: 550 FP, likely 850 FP, Max 1300 FP
Implicit likely expert PDR: 5.500/850 = 6,5 h/FP
Selecting the most relevant projects in the ISBSG D&E repository
show the next results:
PDR (h/FP)
Min. 3,2
Percentile 10% 4,3
Percentile 25% 6,2
Median 8,9
Percentile 75% 12,9
Percentile 90% 19,8
Max. 34,2
N 89
PDR (h/FP)
Min. 3,2
Percentile 10% 4,3
Percentile 25% 6,2
Median 8,9
Percentile 75% 12,9
Percentile 90% 19,8
Max. 34,2
N 89
550 850 1300
3.410 5.270 8.060
4.895 7.565 11.570
7.095 10.965 16.770
Functional Size
550 850 1300
3.410 5.270 8.060
4.895 7.565 11.570
7.095 10.965 16.770
Functional Size
5.500 hours
seems optimistic
15. ISBSG Reality Check: Duration
Same analysis is possible
Also, formulas have been published in the Practical Project
Estimation book
For instance:
table C-2.2 Project Duration, estimated from software size only
Functionele omvang 550 FP
C uit tabel 0,507
E1 uit tabel 0,429
Duration = C * Size^E1
Duration = 7,6 elapsed months
550 850 1300
Duration 7,6 9,2 11,0
16. Result
Expert estimate was assessed optimistic
Adjusted Estimate:
Effort: 8.000 hours
Duration: 10 months
This turned out to be quite accurate!
The project manager now always carries out reality
checks and is ‘spreading the word’.
18. Case 2
Senior management of a software company wondered how
competitive they were when it comes to productivity.
Many bids for projects were lost and they wished to improve,
especially their Microsoft .Net department.
Analysis of the bids by department showed the next figures:
Nr. of bids 23
Average PDR in bid 16,3 h/FP
Average Size (FP) 230 FP
Average teamsize 6 fte
PDR (h/FP)
Min. 3,2
Percentile 10% 3,8
Percentile 25% 5,9
Median 7,6
Percentile 75% 12,9
Percentile 90% 18,9
Max. 34,2
N 35
ISBSG data
analysis
19. Issues
Analysis of the bid phase showed a number of issues:
Estimates were extremely pessimistic due to severe
penalties in case of overruns;
In a number of stages, risk surcharges were added;
They wished to work in fixed team of 6 fte, but ISBSG data
shows that the project size was usually too small for this
teams size to be efficient;
Because of the knowledge that the department bids were not
market average (or better), the bid process was redesigned,
making the company more successful!
21. Case 3
An organization decided to outsource all of their IT
work to one supplier;
Therefore, the internal competition has been gone and
potentially the chosen supplier could charge whatever
they wish;
The ISBSG data was used to assess the productivity
and competitiveness of the suppliers;
After the bidding procedure, the chosen supplier had
to commit to productivity targets and quality targets.
22. Bidding process
5 ICT suppliers bid for the 5-year contract;
They had to submit data of 6 completed projects, sized in
IFPUG function points;
The company metrics desk (MD) assessed the productivity
and competitiveness of the organizations;
The MD constructed a model to do the assessment;
The model was part of the overall bid process and an
important part of the selection of the one winner;
The data submitted was also the basis for the baseline
performance that he had to live up to.
23. Assessment model
Reality value;
Compliancy value to data requirements;
Productivity / Quality value.
Project Delivery Rate (PDR) = spent project effort related to
function point (h/FP);
Productivity Index (PI) = metric from QSM, derived from size,
duration and effort;
Quality: delivered defects per FP;
Benchmarks:
• PI against the QSM Business trend line
• PDR against ISBSG Repository
• Adjusted = normalised to Construction+Test activities
24. Productivity/quality assessment
Supplier PIscore
Rank
PIscore
Points
PIscore
Supplier A 3,9 2 8
Supplier B 5,0 1 10
Supplier C 3,4 3 6
Supplier D 3,0 5 2
Supplier E 3,2 4 4
Supplier
PDR
score
Rank
PDR score
Points
PDR score
Supplier A -3,2 1 10
Supplier B -2,1 2 8
Supplier C 16,6 4 4
Supplier D 6,2 3 6
Supplier E 18,3 5 2
Supplier
Quality
Score
Rank
Quality score
Points
Quality score
Supplier A 3,1 1 10
Supplier B 13,9 2 8
Supplier C 52,6 3 6
Supplier D 1000,0 5 2
Supplier E 94,6 4 4
Supplier
Points
PIscore
Points
PDR score
Points
Quality score
Productivity/
Quality value
Supplier A 8 10 10 9,0
Supplier B 10 8 8 9,0
Supplier C 6 4 6 5,4
Supplier D 2 6 2 3,2
Supplier E 4 2 4 3,4
weight 50% 30% 20%
26. Supplier targets and management
7
8
9
10
11
12
13
PDR(h/FP)
Target PDR PDR(h/FP)
500
600
700
800
900
1000
1100
1200
1300
EUR/FP
Target PCR PCR (EUR/FP)
0
5
10
15
20
25
30
35
40
45
jan-13 feb-13 mrt-13 apr-13 mei-13 jun-13 jul-13 aug-13 sep-13 okt-13
Target Defects/1000 FP Defects/1000 FP
The supplier is measured continuously
and still has to make his target for the
first time!
The organization is happy that the
trends show improvement and they feel
in control.
27. Other useful purposes of ISBSG data
Analyze the difference in productivity or quality
between two (or more) types of projects:
Traditional vs. Agile
Outsourced vs. In-house
Government vs. Non-government
One site, multi site
Reuse vs. no reuse
ISBSG Special Analysis reports
28. Special reports, examples
Impact of Software Size on Productivity
Government and Non-Government Software Project
Performance
ISBSG Software Industry Performance report
ISBSG/COSMIC The Performance of Business Application, Real-
Time and Component Software Projects
Estimates – How accurate are they?
Planning Projects – Role Percentages
Team size impact on productivity
Manage your M&S environment – what to expect?
Many more