1

Online Student Guide

OpusWorks 2019, All Rights Reserved

Measurement System Analysis

2

Table of Contents

LEARNING OBJECTIVES ........................................................................................................................................ 4

KEY DEFINITIONS .................................................................................................................................................. 4

INTRODUCTION ...................................................................................................................................................... 4 COLLECTING DATA EXERCISE ................................................................................................................................................ 5 CATEGORIES ............................................................................................................................................................................. 5 MEASUREMENT ........................................................................................................................................................................ 6 ACCURACY ................................................................................................................................................................................. 7

BIAS, LINEARITY AND STABILITY .................................................................................................................... 7 BIAS ........................................................................................................................................................................................... 8 LINEARITY ................................................................................................................................................................................ 8 STABILITY ................................................................................................................................................................................. 9 PRECISION ................................................................................................................................................................................. 9

GAUGE R&R ........................................................................................................................................................... 10 REPEATABILITY ..................................................................................................................................................................... 10 REPRODUCIBILITY ................................................................................................................................................................. 10

VARIABLES AND ATTRIBUTE DATA ............................................................................................................. 11 ARE THESE PROCESSES THE SAME? ................................................................................................................................... 11 GAUGE R&R STUDY .............................................................................................................................................................. 12 GAUGE R&R: PRECISION ...................................................................................................................................................... 12 MEASUREMENT SYSTEM METRICS ..................................................................................................................................... 13 GASKET EXAMPLE STEP 1-3................................................................................................................................................ 14 GASKET EXAMPLE STEPS 4-6.............................................................................................................................................. 14

GRAPHS & TABLES.............................................................................................................................................. 15

MINITAB OUTPUT ............................................................................................................................................... 16 GAUGE R&R % CONTRIBUTION ......................................................................................................................................... 16 P/TV RATIO ........................................................................................................................................................................... 17 P/T RATIO .............................................................................................................................................................................. 17 NUMBER OF DISTINCT CATEGORIES ................................................................................................................................... 18

ATTRIBUTE STUDY ............................................................................................................................................ 18 GAUGE EFFECTIVENESS % ................................................................................................................................................... 20 REPEATABILITY ..................................................................................................................................................................... 21 GAUGE PRECISION % ............................................................................................................................................................ 21 GAUGE SUMMARY .................................................................................................................................................................. 22 TOTAL VARIATION ................................................................................................................................................................ 22

RULES OF THUMB ............................................................................................................................................... 23 GUIDELINES ............................................................................................................................................................................ 23

3

© 2019 by OpusWorks. All rights reserved. August, 2019 Terms of Use This guide can only be used by those with a paid license to the corresponding course in the e-Learning curriculum produced and distributed by OpusWorks. No part of this Student Guide may be altered, reproduced, stored, or transmitted in any form by any means without the prior written permission of OpusWorks. Trademarks All terms mentioned in this guide that are known to be trademarks or service marks have been appropriately capitalized. Comments Please address any questions or comments to your distributor or to OpusWorks at info@OpusWorks.com.

4

Learning Objectives

Upon completion of this course, students will be able to: • Identify the characteristics of a good measurement system • Identify the benefits of using a Gauge R&R study to validate the measurement system • Discuss the steps used to conduct a Gauge R&R study • Use the results of the Gauge R&R study to determine how effective the measurement system is

Key Definitions

Accuracy is a measurement of how close the average of multiple measurements of an event are equal to the true value. Precision is a measurement of the variation in repeated measurements of the same event. Linearity measures how well the measurement system maintains its performance over a range of events. Stability measures how well the measurement system maintains its performance over time. Gauge repeatability is a measure of how consistently a measurement system measures the same event over time. Gauge reproducibility is a measure of how consistently several operators or measurement systems measure the same event over time.

Introduction

The project team identified the customer’s critical to quality characteristics and began identifying the process variables, or vital X’s, that had a direct effect on the CTQ variable. One of the team’s primary responsibilities during the Measure phase is to validate the process for collecting and analyzing the data required to improve the process.

5

Collecting Data Exercise

Let’s begin by looking at collecting data. You are assigned the task of reducing the number of calls handled by associates at a call center due to balance inquiry. This Pareto Chart shows the frequency of requests made by customers during calls received on Monday. After sharing the results with other team members, there were some concerns that “balance inquiry” could be ranked number one. To verify the results, you personally collected data on Tuesday and produced the Pareto Chart shown here. Which of the following do you think might account for the difference in the number of calls coded as balance inquiry on Monday versus Tuesday?

Categories

There are three general categories we can use to classify the reasons accounting for the change in the data just observed.

The first is Expected Variation. As you have previously learned, everything varies. Simple day-to-day variation in the data can be expected and may be attributed to common cause variation. The next category is related to a change in the process. Process changes can have a significant impact on the analysis of the data collected from the process. The third category is a change to the measurement system. For example, the variation could be caused by changes in specifications or procedures for collecting the data, the use of multiple gauges for measuring, and differences from employee to employee involved in data collection. With business process improvement, you will be making the transformation to a data-driven culture where objective decisions will be based upon data. A top priority must be to ensure the validity and the reliability of the measurement process.

6

Measurement

All work can be viewed as a series of interconnected processes with inputs and outputs. During the development of the process map, the team will define the characteristics that need to be measured from the inputs and outputs of the various stages of the process. This information becomes the inputs to the measurement process.

Too often, people think of the measurement process as just the computers that collect the data and issue reports. While the information system used to collect the data is an essential part of the measurement process, the measurement process must also include the process of assuring the measurements themselves are accurate and precise.

These measurements or data are the outputs of the measurement process. The analysis of this data will produce the information used to make critical business decisions. Let’s look at the following simple measurement process and discover the four characteristics that a good measurement process should have.

7

Accuracy

If you took a survey and asked, “What is a characteristic that a good measurement system must have?” the number one answer might be, “Accuracy”. But if you then asked for the meaning of accuracy, you might be surprised by the different interpretations you might hear. In the system shown here, we shoot arrows at a defined target and measure the distance from the center. The target has a value of ZERO. Arrows that land to the left of center are coded as negative numbers and the arrows to the right are positive numbers. In Process A, the arrows landed at +5 and –5, so our measurements are +5 and –5. In Process B, the arrows landed closer to the target at +1 and –1, so our measurements are +1 and –1.

If we define the accuracy of a process to be “the average value of multiple measurements of an event is equal to the true value,” then which process is more accurate?

Bias, Linearity and Stability

The accuracy of the measurement system has three components: bias, linearity, and stability. Bias ensures that the measurement system maintains its performance to standards or master values. Linearity measures how well the measurement system maintains its performance over a range of events. And stability measures how well the measurement system maintains its performance over time.

8

Bias

A lack of accuracy due to bias may be affected by two different sources. Lack of good calibration to a true standard will cause all measurements to be off by some amount. This could be due to incorrect or outdated calibration. Also, different operators or machines are centered differently, such that their average is offset from the true value. In our target example the true standard is the bull’s-eye. If the arrow is aligned or sighted improperly when the shooter aims then the arrow will consistently miss the target.

Linearity

Linearity is a measure the difference in accuracy or precision over the range of the instrument capability and can be measured with the Scatter Plot of true value versus measured value. In this example both Gauge 1 and Gauge 2 have a problem with linearity. Gauge 3 does not have a linearity problem. Using our target analogy linearity would answer the question “can we hit the target accurately at close distances and the long distances?”

9

Stability

Stability can also affect both accuracy and precision. It is a measure of how well the system performs over time. Stability can be evaluated using a trend charter a run chart and maintained through a regular calibration cycle, additional Gauge R&R studies and a process monitoring program. In our target example, if we were to shoot arrows all day, would we have the same result throughout the day?

Precision

Accuracy alone can not be the only factor in determining a good measurement system. A good measurement system must also be very precise. That means, there should be little variation in repeated measurements of the same event. In the system shown here, “Which process is more precise?”

10

Gauge R&R

Precision, or measurement variation, has two components: Repeatability is a measure of how consistently a measurement system measures the same event over time. Reproducibility is a measure of how consistently several operators or measurement systems measure the same event over time. The study of the repeatability and reproducibility of a measurement system is called Gauge R&R. We'll discuss Gauge R&R in more detail later in this module.

Repeatability

To measure gauge repeatability, we repeatedly measure the same event with the same measurement system. Since the event does not change, any change in the measurements must be due to changes in the measurement system. In this example, the same operator will measure the process at different times using the same measurement system. We are measuring how repeatable the system is.

Reproducibility

To measure gauge reproducibility, we have several people or systems repeatedly measure the same event. We look for differences in the results between the people or the system. In this example, different operators will measure the process at different times using the same measurement system. We are measuring if these two operators can produce the same results using the same system.

11

Variables and Attribute Data

Now that we understand the basics and terminology of measurement systems, let’s look at the two types of measurement systems analysis, one for variables or continuous data which is called a Gauge R&R study. The other is for attribute or discrete data and it is called an attribute agreement analysis. Depending on the type of data, the statistical analysis will be different.

Are These Processes the Same?

We make decisions about process changes and parts’ conformance to specifications which are based on measurements or inspection. If the data we collect is accurate we will make the right decisions. As we will see, the measurement system will have a strong influence on data accuracy. Here are two process with the same total observed variability which is a combination of the true process variability plus variability due to the measurement system. Choose a process which you feel has the best potential to provide accurate data.

12

Gauge R&R Study

A Gauge R&R study is the method used to assess the variability of the measurement system compared to the expected part-to-part variability in the process. Measurement system variability can be divided into 2 components; variation due to the gauge or equipment which is called repeatability, and variation due to operators which is called reproducibility. Gauge R&R is a measure of the repeatability and reproducibility of the measurement system.

Gauge R&R: Precision

In other words, it is a measurement of the precision of the measuring system. To calculate Gauge R&R we will be looking at both variances and standard deviations of the data. Recall that the standard deviation is the square root of the variance but standard deviations cannot be added and must be converted to variances in order to perform the needed calculations. Gauge R&R is the square root of two components of variation – the variance for repeatability plus the variance for reproducibility.

13

Measurement System Metrics

To determine whether the measurement system is “good” or “bad” for a certain application, we will compare the variation due to the measurement system to the total process variation and to the tolerance, or the product specification which is expressed as the upper specification limit minus the lower specification limit. There are 4 metrics for a Gauge R&R study: The first is the percent contribution of Gauge R&R variance to the total observed variation. In a good measurement system this contribution will be 1% or less. 1 to 9% indicates a marginal system, and greater than 9%, an inadequate system which must be fixed or replaced. The second metric is the percent study variation which is called the P to TV (Precision to Total Variation) ratio. Where the Gauge R&R metric compared variances, the P to TV ratio looks at standard deviations. A good P/TV ratio is less than 9%, marginal is 9 to 30 %, and an inadequate system is greater than 30%. The third metric is the precision to tolerance ratio, called P to T ratio. This compares the standard deviation of the measurement system to the tolerance, or to the upper specification limit minus the lower specification limit. It shows the percent of the tolerance range consumed by measurement system variation. It is used when evaluating parts with respect to the specifications. The P/T ratio has the same acceptance criteria as the P/TV ratio. The final metric is the number of distinct categories which estimates how many separate groups of parts the measurement system can distinguish. This number needs to be as large as possible and equal to or greater than 5. To show the steps for completing a Gauge R&R study and the calculations for these metrics, we will use an example.

14

Gasket Example Step 1-3

In this example we will be measuring the thickness of a gasket which has a specification of 16 plus or minus 0.25 mm. A Gauge R&R study consists of several parts that are repeatedly measured by multiple appraisers. While the numbers can vary, most studies use 10 parts and three appraisers who measure the parts at least two times each. The first step is to collect at least 10 samples that represent the full range of long-term process variation and identify the samples in a way that the operators will not know which item they are measuring. In addition, identify the operators who use this instrument or measuring system daily. The second step is to ensure that the gauge is properly calibrated or verify that the last calibration date is valid. The third step is to set up data collection with the columns shown here.

At a minimum you must capture a part ID, operator designation, the number of trials, and a column for the measurements. In our case we will be measuring the thickness of 10 gaskets with three operators and two trials each.

Gasket Example Steps 4-6

For the 4th step we will ask the first operator to measure all the samples once in random order. Blind sampling, in which the operator does not know the identity of each part, should be used to reduce human bias. Fifth we will have the second operator measure all the samples once in random order and continue until all operators have measured the samples once (this is Trial 1). And finally we will repeat steps 4 & 5 for the required number of trials. The Minitab command and dialog boxes are shown here.

15

Graphs & Tables

The output of a Gauge R&R study will be presented in the form of graphs and tables. The first draft is a bar chart of the variation due to Gauge R&R and part-to-part variation. Data also appears for the repeatability and reproducibility elements. A desirable measurement system will have the majority of the variation in the parts being measured and very little in the Gauge R&R element as shown here.

The second graph is a range chart by operator which indicates the consistency of the measurement system. Points above the control limit indicated parts which were measured inconsistently and should be investigated. The third graph is an X-bar chart by operator. This is a slightly different application of an X-bar chart and is interpreted differently from an X-bar chart used for process control. In this case the control limits represent the repeatability of the measurement system. Think of the distance between the control limits as the variation or noise of the measurement system. A good measurement system, such as the one shown here, will have at least 50% of the measurements outside this range.

16

Minitab Output

These tables which are a typical output of statistical software show the variances broken down into separate components and measured for the percent they contribute to the total variation of the process. Minitab output is shown here. Note: sometimes 'Gage R&R' is shown in an alternative spelling as it is here in this Minitab output. Both are acceptable.

Gauge R&R % Contribution

In this table the variance of the process is broken into several components so you can assess the contribution of each to the total variation. The % contribution of Gauge R&R addresses what percent of the Total Variation is due to measurement error. In this example the total Gauge R&R is 5.24% of the total observed variation and the part-to-part variation is 94.76%. This is calculated by dividing the variance component or total Gauge R&R by the variance component for total variation. This percentage falls in the marginal range and at this process should be improved based on criticality and cost. You can also see that repeatability accounts for 4.02 percent and reproducibility, or operator-to-operator variability is 1.22 percent or the total. Any improvements should target the measuring equipment and not the operators.

17

P/TV Ratio

The percent study variation addresses what percent of the total variation is taken up by measurement error. This metric compares the measurement system variation to the total variation. A study variation is defined as six times the sample standard deviation obtained from the data and represents the full spread of a normally distributed process. This value of 22.89% also falls in the marginal category. Like the Gauge R&R, this is calculated by dividing the study variation for each component by the total variation.

Note that in this case the percentages do not add up to 100%. This metric is looking at standard deviation rather than variance because the standard deviation uses the same units as the part measurements and tolerance and allows for more meaningful comparisons.

P/T Ratio

The precision/tolerance ratio is a measure of how much of the tolerance is being consumed by measurement system variation. In our example the specification is plus or minus .25 mm for a tolerance of .5 mm. The generally accepted goal for the P/T ratio less (10%). In this example, the measurement system variation consumes 36.1 percent of the product tolerance, which places it into the unacceptable range. If your question is "Is this gauge adequate to inspect a part to this tolerance?", then P/T ratio answers this question. The typical standard for P/T ratio is 10% or less is ideal, between 10% to 30% may be

18

acceptable depending on the importance of the measurement, the cost of new gauging, and so on, and greater than 30% is not acceptable.

Number of Distinct Categories

This metric defines the number of distinct categories, which could be established with the measurement system in the range of process/product variation while making allowance for measurement error. For example, if the number of distinct categories is two then the measurement system is only able to distinguish high from low. If this value is 5 or greater then the measurement system is more sensitive. In this case 6 distinct categories is acceptable.

Attribute Study

To illustrate some of the concepts just learned, we are going to look at an example of a measurement system study using discrete data. This is known as an Attribute Agreement Analysis. In this study, 30 cards were dropped to the floor and videotaped with a camera above the target. Three operators were chosen at random from the pool of active operators. Each operator watched the video and decided whether each card drop was a hit or a miss using the definition given for determining a hit. After some time passed, the drops were randomized and the process repeated for each operator. These are called the trials of the attribute agreement analysis. This is the data collected from the study. Let’s examine this data in more detail to help us understand how to interpret the results from our study.

19

First we should notice that there are 30 different data points collected for three different operators on each of the two trials. A closer look at the data will help us understand how to measure the accuracy and repeatability of each operator. Let’s first look at how to calculate a measure of accuracy. Operator Accuracy Score is the percent of time the operator’s answers match the Master Attribute in both trials. The Master Attribute is the established standard or correct answer. We will use the data from Operator A to illustrate this concept and ask you for help.

For Sample 1, the Master Attribute was a Hit. Remember, the Master Attribute is the defined standard or correct answer. Examine the results for both trials of Operator A, sample 1. Does the response for both trials for Operator A agree with the Master Attribute?

We continued this process for all 30 trials for Operator A and found 23 matches in the 30 samples. Operator A’s accuracy score is 76.7 percent. Similarly, the accuracy score was calculated for the other two operators. Here are the results.

20

Some of the same conclusions you may have thought of are probably illustrated in this graphic. This chart clearly shows the difference among the accuracy scores for the three operators with Operator B having the worst performance. We might also conclude that none of the operators achieved a very high accuracy score. This would lead us to question the overall effectiveness of the measurement system. To measure this, we calculate an overall gauge effectiveness percentage. This is the percent of time ALL operators agree with the Master Attribute.

Gauge Effectiveness %

To calculate the overall gauge effectiveness percentage, we count the number of samples where ALL the operator’s trials agree with the Master Attribute and divide by the number of samples, 30. Let’s start with the first sample. All items shown highlighted in the first sample must match. Here we see that Operator B did not agree. Therefore, we put the letter “N” in the column labeled “Overall Accuracy vs. Attribute”.

We move to sample 2. Here we have agreement among all the operators with the master attribute. For this sample, we put the letter “Y” in the column. This process is continued until we have completed the analysis for all 30 samples. We then count the number of Y’s, which totals 14 and divide by 30 to get an overall gauge effectiveness percentage of 46.7%. This score, 46.7% is much too low. Accuracy is a problem. You cannot have an effective measurement system with a level of accuracy this low.

21

Repeatability

As we have previously observed, besides measuring accuracy, we also need to measure how precise a system is.

Let’s look at a measure of operator repeatability. This is the percent of the time the operator's answers for the two trials are the same. Therefore, we do not need the master attribute data to calculate repeatability. We will use the data from Operator A to illustrate this concept. For Sample 1, both trials were a HIT. This is a match. For sample 2, both trials were a Hit. This is a match. For sample 3, Miss. This is a match even though it

did not agree with the attribute. Remember, we are only measuring the consistency of the operator, not how accurate the operator is. We continue counting the number of matches for all 30 samples for each operator and compute each operator’s repeatability score. The operator repeatability score tells us how consistent each operator is as an individual inspector, but not how consistent the operators are among themselves. To find out this measure we calculate an overall gauge precision percentage.

Gauge Precision %

The overall gauge precision percentage is the number of matches across all operators regardless of the attribute score. Count the number of matches, the number of ‘Y’s” in the column labeled overall gauge precision and divide by 30. Here we see the score is 60%. This number indicates that only 60 percent of the time did all operators agree among themselves. This number represents a level of precision of reproducibility of our measurement system.

22

Gauge Summary

So what have we learned? Let’s summarize some of the more obvious facts: The accuracy level or overall gauge effectiveness percentage of our measurement system is very low at 46.7%.

The overall precision of the measurement system is low at 60%. There is a difference in the level of accuracy among operators, one being much worse than the other two. The level of precision for an individual operator is very high. Individuals are consistent in their calls from trial to trial but not always accurate.

Total Variation

Why is it so important for your measurement system to be accurate and precise? The answer is, it affects the total variation being measured by the system. The variation in the data collected from the process will have two components, process variation and variation due to the measurement system itself. If the variation from the measurement system itself is large, it will be difficult to determine if a problem exists, what are the real causes, and what the true capability of the process is. A top priority of any Process Improvement team must be to determine the validity of the measurement system and if a problem is found, fix it. You should fix the measurement system to reduce perceived output variation. You should fix the process to reduce actual output variation.

23

Rules of Thumb

To help you as you begin to check your measurement system here are some Rules of Thumb for conducting an attribute agreement analysis: You should have at least 30 different data points. You need at least 3 different operators. Operators should be individuals who normally do the process. Randomize the data points to make it as realistic as possible and remove any bias. The master attribute data should have at least 5 of each attribute. For example, 5 good and 5 bad or 5 hits and 5 misses.

Guidelines

If your study shows your measurement system has a problem with accuracy, here are some guidelines to follow: If people are consistent, but wrong in their ratings, then they are misapplying rules or following the wrong measurement process. When this happens it can often be fixed by clearing up: Misunderstanding of procedures, Key definitions.

If your study shows your measurement system has a problem with consistency issues, here are some guidelines to follow:

If people are not consistent in their ratings, then they are generally not following a consistent process to take their measurements. When this happens, look for: Poor tools, Lack of process compliance by the operators. This problem is generally an indication of a poorly designed or unstable measurement process.