Post on 08-Nov-2021
Measurement System Analysis
How-to Guide - Appendices Version 6.1
August 2013
2 | © 2013 Rolls-Royce plc
MSA How-to Guide
© 2013 Rolls-Royce plc
The information in this document is the property of Rolls-Royce plc and may not be
copied or communicated to a third party, or used for any purpose other than that for
which it is supplied without the express written consent of Rolls-Royce plc.
This information is given in good faith based upon the latest information available to
Rolls-Royce plc, no warranty or representation is given concerning such information,
which must not be taken as establishing any contractual or other commitment binding
upon Rolls-Royce plc or any of its subsidiary or associated companies.
© 2013 Rolls-Royce plc | 3
Step 1 Be Prepared
Step 2
Plan the Study
Step 3
Conduct the Study
Step 4 Type of Study
Continuous Data:
Gauge R&R for continuous data
Attribute Data: Attribute agreement
analysis for attribute data
Step 5
Taking action if the results are unacceptable
Step 6 Maintaining the improvement
1
2
3
4
5
6
This appendices provides supplementary information on how to carry out analysis using Minitab statistical software – together with some of the more detailed analysis of the statistical output.
CONTINUOUS DATA: Appendix 1:
Setting up and randomising the
spreadsheet in Minitab
CONTINUOUS DATA: Appendix 2:
Entering the data in Minitab
ATTRIBUTE DATA: Appendix 6: Setting up and randomising the spreadsheet in Minitab
ATTRIBUTE DATA Appendix 7: Entering the data in Minitab
Appendix 3: Carrying out Gauge R&R
in Minitab
Appendix 8: Carrying out Attribute Agreement Analysis in Minitab
Appendix 4: Supplementary Information on
Interpreting the Graphical Output from Gauge R&R in Minitab
Appendix 9: Supplementary Information on Interpreting the Output from Attribute Agreement Analysis
4 | © 2013 Rolls-Royce plc
MSA How-to Guide
1. Setting-up and Randomising the Spreadsheet in Minitab
2. Entering the Data in Minitab
3. Carrying out Gauge R&R in Minitab
4. Supplementary Information on interpreting the output
5. FAQ for Gauge RR
Continuous Data
In This Section:
© 2013 Rolls-Royce plc | 5
Setting up and randomising the worksheet for a Gauge R&R Study for
Variable Data 1. The starting point for setting up the worksheet for a variable data Gauge R&R
study is the same regardless of which randomisation method for the
worksheet is required. To begin go to:
Stat > Quality Tools > Gage Study > Create Gage R&R Study Worksheet
2) Enter the identity of the parts to be used
1) Enter the quantity of parts to be studied
3) Enter the number of people in the study
4) Enter the identities of the people in the study
5) Enter the number of times that each person will check each part
Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab
Continuous Data
2. Complete the dialogue box for the required detail:
6 | © 2013 Rolls-Royce plc
MSA How-to Guide
This gives you 3 options to randomise the worksheet:
3. Click on the options box
Options
At this stage you must decide which randomisation method to use taking into consideration the practicalities of running the experiment and the most economical use
of people’s time.
The different options are each described below.
Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab
Continuous Data
© 2013 Rolls-Royce plc | 7
a. Do not randomise: As it states, this option does not randomise the data. This
option will sequence the parts then the people for each part and provide a run
order column as shown below.
1) Use the ‘Options’ to confirm selection
3) Note the sequence of parts and people
2) Make the selection and click OK will then generate a worksheet for the study
Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab
Continuous Data
8 | © 2013 Rolls-Royce plc
MSA How-to Guide
b. Randomise all runs: This will completely randomise the order that the
measurements are taken in as shown below. This is useful to prevent the
appraisers from memorising their previous measurements and also to reduce the
impact of time related factors. It does however require all of the appraisers to be
present at once which can be impractical in many situations such as where
different shifts are worked.
As a facilitator, it can also be useful to preserve the ‘standard’ (un-randomised)
order by selecting the option ‘Store standard run order in worksheet’.
Following data collection and analysis the standard order can be used to re-sort the
recorded data so that the pattern of collection may give an insight into what happened.
This should only be done if the measurement system analysis study is not clearly
acceptable and in this case can be useful in identifying combinations which were
awkward for the appraisers.
4) Note the sequence of parts and people
3) Make the selection and click OK will then generate a worksheet for the study
1) Use the ‘Options’ to confirm selection 2) Check the box to
include standard order column
Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab
Continuous Data
© 2013 Rolls-Royce plc | 9
c. Randomise runs within operator: This will prevent memory of measurements by
the people undertaking the study but preserves the appraiser sequence. This
enables a study appraiser’s time to be managed as only one appraiser needs to
be present at specified times. As a facilitator, it can also be useful to preserve
the ‘standard’ (non-randomised) order by selecting the option.
This is the most commonly used option;
On completion of Note 4, Minitab returns you to previous dialogue box (Create Gage
R&R Worksheet) then press OK on this screen.
Minitab will now generate the worksheet, you will need to add in your data column and
collect the data before running the study.
4) Note the sequence of parts and people
3) Make the selection and click OK will then generate a worksheet for the study
1) Use the ‘Options’ to confirm selection
2) Check the box to include standard order column
Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab
Continuous Data
10 | © 2013 Rolls-Royce plc
MSA How-to Guide
Maintaining Data Integrity
It is often overlooked that data integrity starts when the data is entered. In statistical
software such as Minitab it is common to see data formatting and entry errors causing
issues.
The two most common issues to be aware of are as follows:
1) Areas of the worksheet have been previously used OR the wrong sort of data
has been entered resulting in the column being in the wrong data format
The wrong type of data format for columns then has the effect of hiding columns that
are expected to be numeric (or vice versa) when conducting an MSA study.
The most common occurrences of this is when a space is added somewhere in the
column or when the letter O is used instead of 0 (zero). In both cases, even if the
typing error is rectified this will change the format of the column from numeric to text
format.
Text format columns can be identified by the addition of a ‘T’ to the column number as
in the example above.
A ‘space’ was typed in and turns the column type to text
Appendix 2: Entering the Data in Minitab
Continuous Data
© 2013 Rolls-Royce plc | 11
2) The second issue is that of human mistakes when entering the data. To guard against this, the facilitator of the measurement study must control the
study to maintain the concentration, time, speed and discipline required to
type/record each data point. In addition to this, it is possible to assist the person entering the data to select the
correct cell by highlighting the line (descriptions and details) of the active entry.
The example shown has used the option ‘randomise runs within operators’ with the
next entry being from appraiser 2 for part identity 9. It can also be very beneficial to record comments when entries are typed. This
additional information can be useful for analysis where the Measurement System is
not acceptable and further investigation is needed.
Data Type Considerations Types of data are very specific to each MSA study. For Gauge R&R studies the data
is variable and has to be the same units of measure as the operating process. For Attribute Agreement Analysis this is attribute data BUT this can be in the format of
whole (count or scale) numbers or text values.
Left click on the row number will highlight the complete row
Appendix 2: Entering the Data in Minitab
Continuous Data
12 | © 2013 Rolls-Royce plc
MSA How-to Guide
Appendix 3: Running the Analysis
Continuous Data
To run the analysis then use the menu commands: Stat> Quality Tools> Gage Study> Gage R&R Study (crossed)
© 2013 Rolls-Royce plc | 13
A dialogue box will appear. Enter the data into the fields as shown below:
1. Minitab will display in
this window, appropriate
elements of the
worksheet for selection,
transfer the columns for
parts appraisers and
measurement data as
shown.
Appendix 3: Running the Analysis
Continuous Data
2. Click against ANOVA in “Method of
Analysis” This should be the default as Minitab
opens the dialogue box, however should
always be checked
3. Then click OK
14 | © 2013 Rolls-Royce plc
MSA How-to Guide
Click on “Options” to enter the tolerance of the characteristic being
measured. For our example this is 0.5mm. Enter 0.5 into the “Upper spec – Lower spec”
Click on “Do not display percentage contribution” & “Do not display percentage study
variation”.
Click “OK” only once. [This simplifies the graphical output to remove graphs not necessary for our
analysis]
Appendix 3: Running the Analysis
Continuous Data
© 2013 Rolls-Royce plc | 15
Click on “Gage Info” to enter the relevant equipment, team and
information for the study. It is also good practice to record the date of the study for future reference
Fill out the details requested Click “OK” then “OK” again.
Appendix 3: Running the Analysis
Continuous Data
16 | © 2013 Rolls-Royce plc
MSA How-to Guide
The Graphical output will appear as below.
Click on Show sessions folder icon to review the numerical
output”
Part-to-PartReprodRepeatGage R&R
200
100
0
Perc
ent
% Tolerance
10 9 8 7 6 5 4 3 2 110 9 8 7 6 5 4 3 2 110 9 8 7 6 5 4 3 2 1
0.10
0.05
0.00
Parts
Sam
ple
Range
_R=0.0307
UCL=0.0789
LCL=0
1 2 3
10 9 8 7 6 5 4 3 2 110 9 8 7 6 5 4 3 2 110 9 8 7 6 5 4 3 2 1
2.00
1.75
1.50
Parts
Sam
ple
Mean
__X=1.819UCL=1.8504LCL=1.7876
1 2 3
10987654321
2.00
1.75
1.50
Parts
321
2.00
1.75
1.50
Appraiser
10987654321
2.00
1.75
1.50
Parts
Avera
ge
1
2
3
Appraiser
Gage name: v ernier caliper
Date of study : 16th A ug 2006
Reported by : HA SF A D
Tolerance:
M isc: Measurement Sy stem A naly sis
Components of Variation
R Chart by Appraiser
Xbar Chart by Appraiser
Measurement by Parts
Measurement by Appraiser
Parts * Appraiser Interaction
Gage R&R (ANOVA) for Measurement
Appendix 3: The Numerical Output
Continuous Data
© 2013 Rolls-Royce plc | 17
Co
mp
on
en
ts o
f Va
riatio
n
In the following appendix the results and interpretation is explained for each of the six
graphs in the Minitab Graphical Output. Please note that all of the statistical analysis and
graphs should be considered before making conclusions for the study and for the potential
actions required. Note also that ‘no action required’ is a possible and valid conclusion.
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies
Continuous Data
The first graph to look at is the “Components of Variation” on the top right
of the graphical output.
This graph shows where most of the variation in the study came from. The Gauge R&R column shows the % Tolerance taken up by the
measurement system variation. Remember this was 76.99%
If the Part-to-Part columns are high (or very high compared to the others)
this tells us that most of the variation in the study was due to the fact that
the parts being measured were not identical (which we would expect).
If the Repeat columns are high compared to the others, this indicates that
there is a problem with Repeatability (i.e. one or more of the appraisers is
inconsistent with themselves). The remaining graphs will help us
investigate this further.
If the Reprod columns are high compared to the others, this indicates that
there is a problem with Reproducibility (i.e. some of the appraisers are
inconsistent with each other). The remaining graphs will help us
investigate this further.
The Gauge R&R column is the variation component total for Repeat and
Reprod.
In cases such as this example where a problem is identified with the
repeatability and/or the reproducibility of the measurement system then
the remaining graphs should be examined to investigate further. Where no problem is identified from the analysis of the components of
variation then there is no need to examine the remaining graphs
18 | © 2013 Rolls-Royce plc
MSA How-to Guide
Summary – Components of Variation Graph
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies
Continuous Data
* 30% is a generally used acceptance criteria, however manufacturing standards may have tighter requirements. Be sure to consult the relevant measurement standards for your area. Details are contained within the SABRe Supplier Management System Requirements document.
© 2013 Rolls-Royce plc | 19
Gra
ph
ica
l Data
Next we will look at the R Chart by Appraiser graph. This chart shows the Range of the results for each appraiser for each of the 10 parts.
Refer to SPC “How to” for more information on R charts.
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies
Continuous Data
Interpreting the Graphical Data
For a perfectly consistent measurement system, all of the ranges on
the graph would be zero i.e. each part would be measured the same
giving no (zero) range.
However, it is unlikely that they will all be zero, therefore we use this
chart to help us identify any measurements of concern.
We interpret this graph by saying that any point which is above the
upper red line is worth investigating, as this indicates that the range of
results for that appraiser and part was higher than expected.
So in the case study example we can see that appraiser 2 has a
bigger range than the other appraisers. Julie and the team make note
to ask appraiser 2 if they did anything different from the instructions
given and move on the next graph
20 | © 2013 Rolls-Royce plc
MSA How-to Guide
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies
Continuous Data
Summary – R Chart by Appraiser
© 2013 Rolls-Royce plc | 21
Inte
rpre
tatio
n o
f gra
ph
ica
l ou
tpu
t
If you are not sure on interpreting control limits on an Xbar chart then ask a
local Black Belt to help you choose the most appropriate type of MSA.
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies
Continuous Data
Next Julie and the team look at the Xbar Chart by Appraiser graph –
this is the graph from the case study.
This graph shows the average measurement for each part and
appraiser.
Ideally we want the patterns of the data to be identical for all 3
appraisers. If they are not, we should investigate.
The 2 red lines on the chart are control limits. We would expect all at
least 50% of the points to lie outside the control limits (red lines) on
this chart. This is different and the opposite to the conventional use of
SPC charts.
As we can, even though we do have the majority of parts outside the
control limits we can also see the patterns for each appraiser look
different. This is again an indication of poor Reproducibility.
Julie and the team note the results of this graph and ask each
appraiser what they did to identify differences then move to the next
one.
22 | © 2013 Rolls-Royce plc
MSA How-to Guide
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies
Continuous Data
Summary – Xbar Chart by Appraiser
© 2013 Rolls-Royce plc | 23
Inte
rpre
tatio
n o
f gra
ph
ica
l ou
tpu
t
Next Julie and the team look at the Measure by Part graph – this is the graph from the
case study.
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies
Continuous Data
Interpreting the Graphical Data
So in the case study example we can see that:
This graph shows circles for all measured values of each part,
together with the average values for each part (shown by ‘crossed
circles’.)
The average values (crossed circles) are connected by the straight
lines
The graph allows us to compare how consistent the measurements
for each of the parts were in the study.
If the measurement system is consistent, there should be very little
scatter between the measurements for each individual part (in other
words, the circles for each part should almost be on top of each other
or overlapping).
We interpret this graph by saying that any part for which there is a
noticeably larger spread in the results, might be worth investigating. In this case parts 10, 8, 4 & 5 appear to have greater variation than
the rest of the parts. The team need to consider why these parts were
more difficult to measure? Julie and the team note this finding and
move on to the next graph.
24 | © 2013 Rolls-Royce plc
MSA How-to Guide
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies
Continuous Data
Summary – Measurement by Parts
© 2013 Rolls-Royce plc | 25
Inte
rpre
tatio
n o
f gra
ph
ica
l ou
tpu
t
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies
Continuous Data
Interpreting the Graphical Data
Next we will look at the Measurement by Appraiser graph:
This graph shows a box plot for all measured values of each item,
together with the average measurement for each appraiser (shown by
‘crossed circles’).
The average values (crossed circles) are connected by the straight
lines. If the measurement system is perfectly consistent, we would
expect the average values for the 3 appraisers to be the same – in
which case the connecting lines would be horizontal.
We would also expect the spread of results (boxes and whiskers) for
all 3 appraisers to be the same (however, unlike the previous graph,
we wouldn’t necessarily expect the spread to be small, as the results
for all of the parts are shown against each appraiser).
We interpret this graph by saying that if, for any appraiser, there is a
noticeably larger spread in results, or the average value is noticeably
different from the others, this might be worth investigating.
So in the case study example we can see that appraiser 2 has a
larger spread of result than the other appraisers. We can also see the
average values line is not straight indicating a Reproducibility
problem.
Again Julie and the team note the findings and move onto the final
graph.
26 | © 2013 Rolls-Royce plc
MSA How-to Guide
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies
Continuous Data
Summary – Measure by Appraiser Graph
© 2013 Rolls-Royce plc | 27
Inte
rpre
tatio
n o
f gra
ph
ica
l ou
tpu
t
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies
Continuous Data
Interpreting the Graphical Data
Finally, we will look at the Part*Appraiser Interaction graph:
This graph overlays the average measurements for each item as
measured by each person.
If the measurement system is perfectly consistent, we would expect
all of the lines to be on top of each other so only one line is seen.
Overlaying lines is the ideal situation here, however there can be
occasions when parallel lines occur. This would indicate that the parts
and appraiser interaction is consistent BUT bias between appraisers
exists.
We interpret this graph by saying that if any of the lines is noticeably
separate from the other 2 lines (for one or more of the parts), this is
worth investigating.
Julie and the team agree that this graph confirms some of their
thoughts from the previous graphs. They are now ready to summarise
their findings.
28 | © 2013 Rolls-Royce plc
MSA How-to Guide
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies
Continuous Data
Summary – Part*Appraiser Interaction
© 2013 Rolls-Royce plc | 29
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies
Continuous Data
Use of Gauge Run Charts
One additional graphical tool which can be used to assess differences in measurements between different operators and different parts is a Gauge Run Chart. You can use a gauge run chart in combination with a Gage R&R Study to help determine what is causing the variability in the measuring system. Create a gauge run chart as follows: 1. To begin go to Stat > Quality Tools > Gage Study > Gage Run Chart
2. Complete the dialogue box for the required details
Then click ‘OK’
Enter the columns for
parts appraisers and
measurement data as
shown.
Click on ‘Gage Info’ to
enter the relevant
equipment and study
references
If known the historical process mean
can be entered and will be plotted as
a reference line on the graph
30 | © 2013 Rolls-Royce plc
MSA How-to Guide
Appendix 4: Supplementary Information on interpreting the Graphical Analysis
Output from Gauge R&R Studies
Continuous Data
Use of Gauge Run Charts
The graphical output will appear as shown: This is a plot of all of the observations by operator (denoted by different colours) and by part number (each box numbered 1 – 10 represents one of the 10 parts). The horizontal reference line is the overall mean of the measurements. The plot allows you to see if any patterns are evident in the data. For instance, you might see that one operator consistently measures higher than the others or that the measurements on certain parts vary more when compared to other parts. Here for example you can see that for part 4, appraiser 1 (in black) has measured higher than the other two appraisers. You can also see for part 10 noticeable difference between the measurements of the three appraisers. Looking at the repeatability within appraisers, for parts 2 and 8 it can be seen that appraiser 2 (in red) noticeably differs in their three measurements indicating a repeatability problem.
© 2013 Rolls-Royce plc | 31
Appendix 5: FAQ for Gauge R&R (Typical Manufacturing Questions)
Continuous Data
Q: “Where can I find the acceptance criteria for MSA in manufacturing?” A: “The criteria and other information are contained within SABRe Supplier
Management System Requirements document. Further guidance information on
measurement and inspection is available on the supplier portal, specifically in the Guide
to Dimensional Measurement Equipment document.”
Q: “We only have 2 operators using this gauge, shall I take part in the study to
make the numbers up?” A: “Not unless the operator is trained to use the gauge and familiar with the component
being measured. Having an untrained operator may result in the failure of the study due
to the stability and/or reproducibility of the measurements due to the untrained operator.
It would be far better to compromise on the amount of measurement readings than to
perform a study that is not representative of the way the process works” Q: “We cannot get access to 10 parts, will 5 do?” A: “Lowering the sample size will affect the uncertainty of the test. However there are
times when this may be required. In difficult situations a compromise may be required
but the analyst should be mindful of this when interpreting the data. For example if a
gauge R&R returns 19% of tolerance with only 5 parts there is a reasonable case for
either acquiring further parts for study or asking the operators to repeat the
measurements 4 or 5 times rather than the usual 3” Q: “We can’t get parts that represent the full process variation as the only ones
we have are from the same batch. What should we do?” A: “If the sample does not represent the true process variation then this will affect the
results of gauge R&R against study variance, the number of distinct categories and the
limits on the X bar chart on the R&R output. If this is the case 2 options exist. Either proceed and study the % Gauge R&R against
tolerance.” Q: “Our gauge measures hundreds of features, do I have to run gauge R&R study
on each of them or can I use read-across methodology?” A: “The Quality Management System requires that all product features/characteristics
are measured using capable measurement systems. That said there are situations
where similar features measured with the same gauge presents an opportunity to
demonstrate capability without direct study of every single feature as long as this is
done robustly. This must however be done in a robust and traceable way. Read across
is not permitted on CMM equipment, not because the CMM’s tend to be incapable but
because of the risk of program errors due to the manual nature of the program
creation.”
32 | © 2013 Rolls-Royce plc
MSA How-to Guide
Appendix 5: FAQ for Gauge R&R
Continuous Data
Q: “The gauge is automated (there is no operator influence)! What do I do?” A: “First be sure that there really is not any operator influence – for instance if there is
a setup process which is manual then this may lead to reproducibility problems. If the
gauge is completely automated then a study can be performed with only 1 operator.
The gauge R&R study will report only on the repeatability element of the gauge R&R. If
there is the opportunity to study two gauges simultaneously (e.g. 2 CMM’s) these can
be identified within the R&R study to allow the reproducibility due to different
equipment rather than operator.” Q: “I have a surface finish gauge, and it keeps failing R&R. What do I do?” A:”Some gauges are notoriously difficult to perform gauge R&R on. Some gauges are
the best available for a given measurement. In this situation contact a measurement
practitioner or Metrologist.” Q: “Under what circumstances should I repeat the gauge R&R?” A: “A gauge R&R should be repeated whenever the process (either measurement
process or manufacturing process) changes significantly or the part tolerance is
changed. Also when turnover of labour is high” Q: “My gauge R&R against tolerance is very good but I only get 1 distinct
category. Is my measurement process good or not?” A: “It is likely that the parts selected for the study are not representative of the total
process variation. This will affect the %R&R against study variance, the number of
distinct categories and the X bar chart. If the parts are representative then the
measurement process is not adequate for the application of SPC analysis as the
majority of the variation seen will be from the measurement system and not the
underlying process.” Q: “How many decimal places on my gauge shall I use when conducting the
gauge study?” A: “As a minimum you should ensure that the study represents the requirements on the
part drawing but the more the better. For instance if the drawing requires measurement
to 2 decimal places, run the gauge R&R to 3.” Q: “My gauge passes the gauge R&R study. Is this all I need to consider?” A: “No – gauge R&R will not highlight gauge bias. For instance it is possible to be
repeatibly wrong. Comparison with a known standard will enable you to study the
amount of gauge bias. Calibration is not done on production parts, real parts can
introduce large differences.”
© 2013 Rolls-Royce plc | 33
6. Setting-up and Randomising the Spreadsheet in Minitab
7. Entering the Data in Minitab
8. Carrying out Attribute Agreement Analysis in Minitab
9. Supplementary Information on how to interpret the output
Attribute Data
In This Section:
34 | © 2013 Rolls-Royce plc
MSA How-to Guide
Attribute Agreement Analysis Worksheet Configurations
1. Setting up the worksheet for an attribute data, Attribute Agreement Analysis is
very similar to a Gauge R&R study where, the worksheet construction all start
from an identical point independent of which randomisation for the worksheet is
used. This is found at:
Stat > Quality Tools > Create Attribute Agreement Analysis Worksheet
Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab
Attribute Data
© 2013 Rolls-Royce plc | 35
2. Complete the dialogue box for the required detail:
3) Enter the identity of the parts to be used and the text standard
2) Enter the quantity of parts to be studied
4) Enter the number of people in the study
5) Enter the identities of the people in the study
6) Enter the number of times each person is to inspect each part
Dropdown Selection Sample Standard/attribute unknown: This provides a worksheet very similar to that of a
Gauge R&R study but does need to have an additional column manually added so
that a ‘standard’ agreement can be compared to it. This is the most flexible option. Sample Standard/attribute in text: This constructs the worksheet with the additional
column of text ‘standards’ for comparison, i.e. When comparing judgements such as
good or bad against the standard which is also stated as good or bad. This is the
option used in the following pages. Sample Standard/attribute in numbers: This constructs the worksheet with the
additional column of numerical ‘standards’ for comparison, i.e. When judgements are
made using a scale (often 1 to 5 or 1 to 10), and the standard should be an exact
match. Sample Standard/attribute in worksheet: This favours manual lists already in the
worksheet and provides selection of those column references for the study.
1) There are 4 drop down selections here (see below)
Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab
Attribute Data
36 | © 2013 Rolls-Royce plc
MSA How-to Guide
This gives you 3 options to randomise the worksheet:
3. Click on the options box
Options
Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab
Attribute Data
© 2013 Rolls-Royce plc | 37
1) Use the ‘Options’ to confirm selection
3) Note the sequence of parts, people and the ‘standard’
2) Make the selection and click OK to generate a worksheet for the study
a. Do not randomise: As it states, this option does not randomise the data. This
option will sequence the parts then the people for each part and provide a run
order column as shown below.
Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab
Attribute Data
38 | © 2013 Rolls-Royce plc
MSA How-to Guide
4) Note the sequence of parts, people and the ‘standard’
3) Make the selection and click OK to generate a worksheet for the study
1) Use the ‘Options’ to confirm selection
2) Check the box to include standard order column
b. Randomise all runs: This will completely randomise the order that the
measurements are taken in as shown below. This is useful to prevent the
appraisers from memorising their previous measurements and also to reduce the
impact of time related factors. It does however require all of the appraisers to be
present at once which can be impractical in many situations such as where
different shifts are worked.
As a facilitator, it can also be useful to preserve the ‘standard’ (un-randomised)
order by selecting the option ‘Store standard run order in worksheet’.
Following data collection and analysis the standard order can be used to re-sort the
recorded data so that the pattern of collection may give an insight into what happened.
This should only be done if the measurement system analysis study is not clearly
acceptable and in this case can be useful in identifying combinations which were
awkward for the appraisers.
Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab
Attribute Data
© 2013 Rolls-Royce plc | 39
4) Note the sequence of parts, people and the ‘standard’
3) Make the selection and click OK to generate a worksheet for the study
1) Use the ‘Options’ to confirm selection
2) Check the box to include standard order column
c. Randomise runs within operator: This will prevent memory of measurements by
the people undertaking the study but preserves the appraiser sequence. This
enables a study appraiser’s time to be managed as only one appraiser needs to
be present at specified times. As a facilitator, it can also be useful to preserve
the ‘standard’ (non-randomised) order by selecting the option.
This is the most commonly used option.
Appendix 6: Setting-up and Randomising the Spreadsheet in Minitab
Attribute Data
40 | © 2013 Rolls-Royce plc
MSA How-to Guide
Maintaining Data Integrity It is often overlooked that data integrity starts when the data is entered. In statistical
software such as Minitab it is common to see data formatting and entry errors causing
issues.
The two most common issues to be aware of are as follows:
1) Areas of the worksheet have been previously used OR the wrong sort of data
has been entered resulting in the column being in the wrong data format
The wrong type of data format for columns then has the effect of hiding columns that
are expected to be numeric (or vice versa) when conducting an MSA study.
The most common occurrences of this is when a space is added somewhere in the
column or when the letter O is used instead of 0 (zero). In both cases, even if the
typing error is rectified this will change the format of the column from numeric to text
format.
Text format columns can be identified by the addition of a ‘T’ to the column number as
in the example above.
A ‘space’ was typed in and turns the column type to text
Appendix 7: Entering the Data in Minitab
Attribute Data
© 2013 Rolls-Royce plc | 41
2) The second issue is that of human mistakes when entering the data. To guard against this, the facilitator of the measurement study must control the
study to maintain the concentration, time, speed and discipline required to
type/record each data point. In addition to this, it is possible to assist the person entering the data to select the
correct cell by highlighting the line (descriptions and details) of the active entry.
The example shown has used the option ‘randomise runs within operators’ with the
next entry being from appraiser 2 for part identity 9. It can also be very beneficial to record comments when entries are typed. This
additional information can be useful for analysis where the Measurement System is
not acceptable and further investigation is needed.
Data Type Considerations Types of data are very specific to each MSA study. For Gauge R&R studies the data
is variable and has to be the same units of measure as the operating process. For Attribute Agreement Analysis this is attribute data BUT this can be in the format of
whole (count or scale) numbers or text values.
Left click on the row number will highlight the complete row
Appendix 7: Entering the Data in Minitab
Attribute Data
42 | © 2013 Rolls-Royce plc
MSA How-to Guide
To run an Attribute Measurement System Analysis (MSA) then use the menu commands:
Stat>
Quality Tools> Attribute Agreement Analysis
Appendix 8: Running the Analysis
Attribute Data
© 2013 Rolls-Royce plc | 43
A dialogue box will appear. Enter the data into the fields as shown below:
Click to confirm data is listed downwards (stacked) This is the format Minitab generates for the worksheet
Enter the column containing the ‘standard’ to be compared to
Only check this box when a scale or multiple class of judgement is used
Appendix 8: Running the Analysis
Attribute Data
44 | © 2013 Rolls-Royce plc
MSA How-to Guide
It is good practice to use the ‘Information’ button to record details about the MSA study directly into the displayed results – click on Information button
The information dialogue will appear, so that you can complete the details as appropriate.
Once completed, click OK button to close the information screen, then click OK again to close on main window.
Appendix 8: Running the Analysis
Attribute Data
© 2013 Rolls-Royce plc | 45
Appendix 8: The Graphical Output
Attribute Data
The Graphical output will appear as below.
Click on Show sessions folder icon to review the numerical output”
46 | © 2013 Rolls-Royce plc
MSA How-to Guide
Appendix 8: The Numerical Output
Attribute Data
Click on Show graphs folder icon to return to the graphical output.
© 2013 Rolls-Royce plc | 47
Click on Show worksheet folder icon to return to the worksheet
Appendix 8: Running the Worksheet
Attribute Data
48 | © 2013 Rolls-Royce plc
MSA How-to Guide
Attribute Agreement Analysis Worksheet Configurations
1. Setting up the worksheet for an attribute data, Attribute Agreement Analysis is
very similar to a Gauge R&R study where, the worksheet construction all start
from an identical point independent of which randomisation for the worksheet is
used. This is found at:
Stat > Quality Tools > Create Attribute Agreement Analysis Worksheet
Attribute Data
Appendix 9: Supplementary Information on interpreting the Output of Attribute
Agreement Analysis
© 2013 Rolls-Royce plc | 49
Interpretation
of graphical
output
The ‘blue’ dots have been explained as the actual proportion of agreement within the
sample of parts appraised in the study. Also shown are the ‘red’ lines which end in a
cross. These indicate the confidence intervals for each Appraiser and for each
Appraiser vs. Standard.
The actual numbers for each confidence interval are recorded in the session window
and used in plotting this graph.
These confidence intervals take into account the fact that the actual agreement %
calculated (as represented by the blue dot) is based only on a relatively small sample
of data. If it was possible to know the ‘true’ agreement % of the appraiser (based on
every part they ever inspected) then this % would be likely to be different from the %
seen in the study sample. The confidence interval indicates the possible range of
values that the ‘true’ % agreement could be. Minitab defaults to a confidence level of
95%. This means that we can interpret the confidence interval for Appraiser 1 for
example as saying “I am 95% confident that the true % of within appraiser agreement
for appraiser 1 is between 62.1% and 96.8% (which is the range indicated by the blue
crosses and red line).
In addition to the ‘How to guide: Measurement System Analysis’ the following pages
give supplementary information on some of the statistical concepts to deepen your
understanding of how to fully interpret the output
Appendix
6
Attribute Data
Appendix 9: Supplementary Information on interpreting the Output of Attribute
Agreement Analysis
50 | © 2013 Rolls-Royce plc
MSA How-to Guide Attribute Data
Appendix 9: Supplementary Information on interpreting the Output of Attribute
Agreement Analysis
Each ‘red’ line and two crosses should be as short as possible indicating less potential
error in the ‘blue’ dot point indication, when using 95% confidence (the default level for
most statistical analysis).
These confidence intervals are calculated using the F distribution which also requires
a thing called degrees of freedom. The larger the degrees of freedom, which can be
influenced by using a larger selection of items for inspection (sample size) and the
quantity of matched agreements (such as Pass / Pass and the standard also as Pass)
also contribute to reducing the confidence intervals.
The F distribution is not symmetrical and therefore the confidence intervals can look a
little odd, one (usually the lower) will be longer than the other.
Interpreting the Lower Confidence Interval It may be desirable in some instances where the quality of the measurement system is
highly critical to use the lower confidence interval (lower red cross) rather than the
observed agreement (blue dot) when assessing the measurement system against the
rules of thumb described on page 46 . This assesses the measurement system based
on the worst case scenario.
You must however take into consideration that the sample size will significantly
influence this. Where the sample size is small (less than 20 parts for example) the
lower confidence intervals will nearly always fall below 70% even when the average
agreement within the sample is fairly good). In these circumstances you should
discuss with a Black Belt an appropriate sample size to use for the assessment of
critical measurement systems. If the measurement needed is highly critical then
consideration should also be given to whether it is possible to redesign the
measurement system to use variable data rather than attribute and assess the system
using Gauge R&R instead.
© 2013 Rolls-Royce plc | 51
The Kappa (correctly quoted as Fleiss Kappa) works on a scale of -1 to 1, where -1
indicates total disagreement between each measurement run and the standard. A
value of 0 (zero) indicates a 50:50 chance of correctly assessing the part which implies
appraisers are ‘guessing’ whether to pass or fail the part. The best possible outcome is
a Kappa value or 1 which indicates total agreement to each round of measures and to
the standard.
Within Appraisers table is shown for a simple Fail or Pass judgement which gives a
Kappa value similar BUT not the same as the percent listing.
Fleiss’ Kappa becomes very useful when a scale or multiple class of judgement is
used which then has a Kappa value indicating agreement for each scale value. An example of this could be shade judgements, where the ends of the scale, shades A
or B (light), I or J (dark) have very high Kappa values near to 1 and middle shades
such as E or F have lower (approximately 0.778291 say) showing for that person these
were harder to judge as correct.
Interpretation
of the
numerical
output
Each of the percentage results shown in the session window tables have been
explained in the ‘How to guide: Measurement System Analysis’. This section will
explain the other statistics such as the ‘Kappa’ listed and the P-value. Each of the
tables is constructed with the same format and therefore the ‘Within Appraisers’ table
will be used for these explanations.
Attribute Data
Appendix 9: Supplementary Information on interpreting the Output of Attribute
Agreement Analysis
52 | © 2013 Rolls-Royce plc
MSA How-to Guide
The P-value (which is short for the probability value) is a back-up statistic describing
the chance of having a Fleiss Kappa near 0 (zero). As the tabular notation shows, the judgement is made about the ‘chance’ (probability)
where the lower the P (vs. > 0) column p-value shows a low number, the less
statistical chance the Kappa values are 0 (zero) or less. In other words, the p-values of 0.0000 are very good and what should be seen.
Interpretation
of the
numerical
output
P-value Judgements
Attribute Data
Appendix 9: Supplementary Information on interpreting the Output of Attribute
Agreement Analysis
© 2013 Rolls-Royce plc | 53
Change History
Revision Date Description of Change Author Owner Approval
V6.1 20/08/2013 Guide reformatted for
SABRe D Prodger D Prodger D Prodger
Document update policy
This document may be updated periodically. Major amendments will be shown as an update from one
revision number to a higher revision number (e.g. revision 1 to revision 2) and therefore the content of
the higher revision will be regarded as the latest requirements. A minor amendment will be shown as a
number change after a decimal point (e.g. revision 1.1 to revision 1.2) and therefore any of these
revisions may be regarded as the latest requirements until a major amendment is introduced
Measurement System Analysis
How-to Guide - Appendices
© Rolls-Royce plc 2013
The information in this document is the property of Rolls-Royce
plc and may not be copied, communicated to a third party or
used for any purpose other than that for which it is supplied,
without the express written consent of Rolls-Royce plc.
While the information is given in good faith based upon the
latest information available to Rolls-Royce plc, no warranty or
representation is given concerning such information, which must
not be taken as establishing any contractual or other
commitment binding upon Rolls-Royce plc or any of its
subsidiary or associated companies.