Decision rule construction requires four components: the measurement result, its measurement uncertainty, the specification limit or limits, and the acceptable level of the probability of making each type of wrong decision

1. What is the
Decision Rule
all about?
ISO/IEC 17025
2005 & 2017
By
Maryam Sourani
Reference: John Wilson
Accreditation and Metrology Services (Pty) LTD
1

2. What is a decision rule
Let’s start with what is a decision?
2

3. What is a decision rule
Let’s start with what is a
decision?
3

4. Years
Where will the next
reading probably be?
4
What is a decision rule
Let’s start with what is a decision?

5. Years
Yes Likely
5
What is a decision rule
Let’s start with what is a decision?

6. Years
Not likely butpossible
6
What is a decision rule
Let’s start with what is a decision?

7. Years
Are these readings in
Specification ?
7
What is a decision rule
Let’s start with what is a decision?

8. Nominal
Are these readings in
Specification ?
Before we decide, we
need to know thelimits
(specification)
8
What is a decision rule
Let’s start with what is a decision?

9. 9
Nominal
So are the readings in
Specification ?
Lower Limit
What is a decision rule
Let’s start with what is a decision?

10. What is a decision rule
• In the current version of ISO/IEC 17025:2005 it states in
clause 5.10.4.2 “When statements of conformance are made,
the uncertainty of measurement shall be taken into account”
• A statement of conformance is something like “The
calibration of this instrument shows that all the readings are
within the
manufacturer’s specifications” or “This instrument complies
with the requirements of the XYZ standard”.
10

11. What is a decision rule
Let’s look at some metrology
Upper Limit
Nominal
On this graph, the uncertainty of
measurement has not been taken
into account.
Lower Limit
11

12. What is a decision rule
Let’s look at some metrology
Upper Limit
Nominal
The Uncertainty bars arenow
included
Lower Limit
John Wilson 2016 12

13. What is a decision rule
Let’s look at some metrology
Upper Limit
Nominal
So are these readingsin
Specification?
Lower Limit
13

14. What is a decision rule
Let’s look at some metrology
Upper Limit
Nominal
Are these readings in
Specification ?
Lower Limit
14

15. What is a decision rule
Nominal
Three different groups
of readings.
Lower Limit
15
Let’s look at some metrology
Upper Limit

16. What is a decision rule
Nominal
So are the readings in
Specification ?
Lower Limit
16
Pas
s
Undecided
• Let’s look at some met rology
Upper Limit
Fail

17. What is a decision rule
Nominal
So does this meet the
requirements of
ISO/IEC 17025?
Lower Limit
17
Pas
s
• Let’s look at some met rology
Upper Limit
Fail
Undecided

18. What is a decision rule
Nominal
The undecided could
however lead to lots
of arguments.
Lower Limit
John Wilson 2016 18
Pas
s
• Let’s look at some meetology
Upper Limit
Fail
Undecided

19. What is a decision rule
Nominal
The undecided could
however lead to lots
of arguments.
Lower Limit
19
Pas
s
• Let’s look at some met rology
Upper Limit
Fail
Undecided
Readings 1 to 4 are clearly “pass” and within
Specification.
Reading 7 is clearly “fail” and outside of
Specification.
It is readings 5 and 6 that cause the arguments.

20. What is a decision rule
Nominal
The undecided could
however lead to lots
of arguments.
Lower Limit
20
Pas
s
• Let’s look at some met rology
Upper Limit
Fail
Undecided
If you are the user, then the undecided are not acceptable
but if you are the supplier you could argue that that they
are acceptable because of the uncertainty associated with
the readings.

21. What is a decision rule
Nominal
The undecided could
however lead to lots
of arguments.
Lower Limit
21
Pas
s
• Let’s look at some met rology
Upper Limit
Fail
Undecided
In this case we have a series of readings over a period of
time so we have an understanding of how the artefact
behaves and can reasonably predict what will happen in
the future. This reduces the risk involved in a decision.

22. What is a decision rule
Nominal
• Let’s look at some metrology
Upper Limit
Similarly – if we had several of the
Lower Limi
same
instruments, such as a production run, and get the above results, we
know a lot more about the behaviour of the products
22

23. What is a decision rule
• Let’s look at some metrology
Upper Limit
Nominal
If we had only one (or one set) of readings, would
we view it any differently?
Is this a pass or fail?
There may be a lot of money or risk
resting on the decision.
23

24. What is a decision rule
• Let’s look at some metrology
Upper Limit
Nominal
Many cal labs would not make a decision based on this
information but would rather pass the information to the
customer to decide for themselves.
So this would not be a statement of conformance
but merely a calibration report.
24

25. • Let’s look at some metrology
Upper Limit
Nominal
If however the lab was tasked to give a statement of
compliance, what would the decision be?
Clearly it depends who’s side you are on.
Different labs could give different decisions.
So the customer needs to be told what “decision rule”
you applied to the statement of compliance.
John Wilson 2016 25
What is a decision rule

26. Decision rules have been used for many years, especially in the calibration and
physical testing industries. Their use has expanded because they are applicable to
any situation in which data is used to make a decision. The terminology for
decision rules includes the following zones and types of zones.
What is a decision rule
26

27. What is a decision rule
In a simple decision rule, the rejection and acceptance zones line up with the specification
zone. The simple decision rule is illustrated in Figure 1. Many compendial monograph
specifications may be considered simple decision rules.
Figure 1: A Simple Decision Rule for a Specification with both Upper and Lower
Limits. If the measurement result lies in the acceptance/specification zone, product is
accepted, otherwise it is rejected.
27

28. A stringent acceptance zone reduces the probability of accepting out-of-
specification product by increasing the risk of rejecting in-specification
product. A guard band zone is established that offsets the specification and
acceptance limits to achieve this. The size of the guard band depends on the
desired probability of making each type of wrong decision. The stringent
acceptance zone can be accompanied by a relaxed rejection zone, which allows
the rejection of product even though a measurement result within the
specification zone by the guard band amount is obtained. Many companies use
such a decision rule and call it “internal release limit.” This decision rule is
illustrated in Figure 2 for a specification with both upper and lower limits and
in Figure 3 for a specification with an upper limit only.
What is a decision rule
28

29. What is a decision rule
Figure 2: A Stringent Acceptance Zone and Relaxed Rejection Zones Decision Rule
for a Specification with both Upper and Lower Limits. Product is accepted if the
measurement result lies within the acceptance zone and rejected otherwise. The
guard band is established by the acceptable probability of making a wrong decision
and the uncertainty.
29

30. What is a decision rule
Figure 3: A Stringent Acceptance Zone and Relaxed Rejection Zone Decision Rule
for a Specification with an Upper Limit Only.
A transition zone decision rule is illustrated in Figure 4 for a specification with an
upper limit only. The transition decision rule should specify the required actions when
a measurement result within the transition zone is obtained. The USP <905>
"Uniformity of Dosage Units" (6) specification is an example of a decision rule that
uses a transition zone. If the acceptance value of the first 10 dosage units is less than or
equal to L1%, the requirements for dosage uniformity are met. If not, additional testing
is performed. The region greater than or equal to L1% can be considered the transition
zone.
30

31. What is a decision rule
Figure 4: A Transition Zone Decision Rule for a Specification with an Upper Limit Only.
Decision rule construction requires four components: the measurement result, its measurement
uncertainty, the specification limit or limits, and the acceptable level of the probability of making
each type of wrong decision. Choice of a decision rule is a business consideration that takes into
account the cost of rejecting an in-specification product, the cost of accepting an out-of-specification
product, the uncertainty associated with the measurement, the distribution of the measurand’s
characteristic, and the cost of making the measurement. The analytical quality-by-design (QbD)
approach uses risk analysis and probability to determine these four components, hence clearly
defining the use of a procedure through a decision rule.
The relationship between these four components allows one to determine if an analytical procedure is
fit-for-use and, furthermore, set acceptance criteria for the analytical procedure to meet. Measurement
uncertainty and its relationship to decision rules will be explored in future columns 31

32. • The current wording of 17025:2005 leaves it very vague
and open to a wide range of interpretation.
• The ILAC G-8:2009 has tried to provide some clarity by
writing this guideline on how to look at pass/fail conformity
assessment.
• The JCGM 106:2012 “Evaluation of Measurement Data- the
Role of measurement uncertainty in conformance assessment
is a further guideline on how to deal with the problem.
32
What is a decision rule

33. • In the current version of ISO/IEC 17025 it
states in clause 5.10.4.2 “When statements of conformance
are made, the uncertainty of measurement shall be taken into
account”
33
What is a decision rule

34. • The ILAC G-8:2009 has tried to provide some clarity by
writing this guideline on how to look at pass/fail conformity
assessment.
• Compliance: If the specification limit is not breached by
the measurement result plus the expanded uncertainty
with a 95% coverage probability, then compliance with the
specification can be stated (See Case 1 of Fig.1). This can
be reported as “Compliance” or “Compliance – The
measurement result is within (or below) the specification
limit when the measurement uncertainty is taken into
account”. In calibration this is often reported as “Pass”;
34
What is a decision rule

35. • The ILAC G-8:2009 has tried to provide some clarity by
writing this guideline on how to look at pass/fail conformity
assessment.
• Non-compliance: If the specification limit is exceeded by
the measurement result minus the expanded uncertainty
with a 95% coverage probability, then noncompliance
with the specification can be stated. (See Case 4 of Fig.1)
This can be reported as “Non-compliance” or “Non-
compliance – The measurement result is outside (or above)
the specification limit when the measurement uncertainty
is taken into account”. In calibration this is often reported
as “Fail”;
35
What is a decision rule

36. • The ILAC G-8:2009 has tried to provide some clarity by
writing this guideline on how to look at pass/fail conformity
assessment.
• If the measurement result plus/minus the expanded
uncertainty with a 95 % coverage probability overlaps
the limit, it is not possible to state compliance or non-
compliance. The measurement result and the expanded
uncertainty with a 95 % coverage probability should then
be reported together with a statement indicating that
neither compliance nor non-compliance was
demonstrated.
• This is what most SANAS labs use today
36
What is a decision rule

37. • The JCGM 106:2012 “Evaluation of Measurement Data- the
Role of measurement uncertainty in conformance assessment”
is a
further guideline on how to deal with the problem.
• This document address a lot more detail and gives different
ways of interpreting results
• This goes into the “knowledge of the
measurand” , Bayes Theorem, conformance probabilities
and much more”
37
What is a decision rule

38. • So the bottom line is that much more knowledge on the
subject is required to enter into an educated discussion with
the
customer about the probability of “False accepts”
when a product should fail and
“False rejects” when a product should pass.
• It is suggested that a good study of JCGM 106:2012 is
made.
• It is available free off the BIPM web site
• There is also a lot more information available
on the Web related to this subject.
38
What is a decision rule

39. • So why all the fuss?
• Again, it depends if you are the user or the supplier and
it boils down to the issue of the probability of “False
Accept” or “False Reject”. What is the risk to both
parties.
• Whilst the current 17025 and the ILAC G- 8:2009
give direction it is not always enforced.
• Very few people know about the JCGM 106.
39
What is a decision rule

40. • So what is proposed in the new 17025 ?
Analysis of the results
7.7.1 Evaluation of conformance
When statement of conformity to a specification or
standard for test or calibration is requested, the laboratory
shall:
a) document the decision rules employed taking into
account the level of risk associated with the decision
rule employed (false accept and false reject and
statistical assumptions associated with the decision
rule employed);
b) apply the decision rule .
NOTE For further information see ISO/IEC Guide 98-4.
40
What is a decision rule

41. • So breaking this down further
When statement of conformity to a specification or
standard for test or calibration is requested, the
laboratory shall:
• The key here is “when it is requested” which implies it is
requested by the customer.
• This means that “contract review” must take place and a
clear definition agreed on BEFORE the job is started.
41
What is a decision rule

42. • During Contract review
The laboratory needs to be prepared to discuss what the
decision rule options are regarding the compliance statement.
They also need to understand what the customer may require and
where the risk of “false accept” or “false reject” lies so that they
do not get drawn into legal battles or compensation claims.
As with all statistical analysis, it is open to different interpretations
and care must be taken to ensure a correct agreement between
parties.
42
What is a decision rule

43. • During Contract review
There are many models which may be applied and they
will vary greatly depending on the application.
You may be testing a new prototype where no previous or
group history is available or you may be checking a
widely manufactured and accepted item against a
manufacturer’s specification. The approach will be very
different.
43
What is a decision rule

44. • Different approaches to each condition
One instrument
over a long time
44
One batch of the
same model
instrument
One reading (or
set of readings on
a prototype
What is a decision rule

45. • Different approaches to each condition
A knowledge of “Guardbands” and the way they
are implemented is advisable.
Upper Limit
Lower Limit
New Limit
Nominal
New Limit
Guardband
45
Guardband
What is a decision rule

46. • Different approaches to each condition
Upper Limit
Guardband
Limit - upper
Nominal
Guardband
Limit - lower
Lower Limit
46
What is a decision rule

47. • Different approaches to each condition
• For example – if the TAR method is used and a 4:1 TAR is
applicable then the rules are
a) Both the UUT and STD must be working in normal
operating specifications
b) The STD must be calibrated and within the normal
calibration interval
c) The UUT reading must be < 70% of the normal
specification limit of the UUT
(This is a 30% guardband)
47
What is a decision rule

48. • Different approaches to each condition
• Similarly, if the guidance of the ILAC G8 is used, then it
must be clearly understood and stated as such.
• The same will apply to the JCGM 106.
• There are also many ISO/SANS/Industry standards that
define clearly what decision rule must be applied.
48
What is a decision rule

49. • In the Lab
… the laboratory shall:
a) document the decision rules employed taking into account
the level of risk associated with the decision rule employed
(false accept and false reject and statistical assumptions
associated with the decision rule employed);
• Take the time to clearly document what has been agreed and
ensure that lab procedures allow for the correct treatment of
the readings and uncertainties.
49
What is a decision rule

50. • In the Lab
… the laboratory shall:
a) document the decision rules employed taking into account
the level of risk associated with the decision rule employed
(false accept and false reject and statistical assumptions
associated with the decision rule employed);
• Might need to change test procedures to cater for the
decision rule that will be applied. It has to be such that
anybody in the lab will know what is required and what the
limits are. 50
What is a decision rule

51. • In the report
… the laboratory shall:
a) document the decision rules employed taking into account
the level of risk associated with the decision rule employed
(false accept and false reject and statistical assumptions
associated with the decision rule employed);
b) apply the decision rule .
• The report must contain either the decision rule used or a
reference to a standard or agreed decision rule
51
What is a decision rule

52. • In the report
… the laboratory shall:
a) document the decision rules employed taking into
account the level of risk associated with the decision rule
employed (false accept and false reject and statistical
assumptions associated with the decision rule employed);
b) apply the decision rule .
• The appropriate statement of conformity which has been
requested or required by a standard.
52
What is a decision rule

53. • In the report
• There must be enough detail so that if another lab is tasked
with taking the same measurements and using the same
decision rule, that they will come to the same
conclusion about a “pass” or “fail”
53
What is a decision rule

54. • In general
• This means that if a certificate or statement of conformance
is required, a lot more detail is required to be addressed
before the measurements are taken and then the Lab will
need to be vey clear about the criteria used and this must be
defined on the test report, certificate of conformance or
statement of compliance.
54
What is a decision rule

55. • In general
• This is not so difficult if it is broken down in
simple logical steps.
• How do you eat an elephant ?
• One bite at a time!
55
What is a decision rule

56. 56
What is a decision rule