**Consumers are increasingly demanding when it comes to the failure-free performance of products and services. Faulty performance can result in loss of sales, claims and/or bad publicity.**

**So for development teams, the risk of developing and introducing a failing product is a major concern. Particularly when the speed of product innovation is high and introduction of new concepts and technology involves risks. **

**The eventual decision to release a newly-developed product is generally based on results from verification tests, which are performed on a limited sample of products to check performance. But how confident can you be, on the basis of this limited sample, that the performance of large volumes of the product as delivered to market will meet specifications?
**

Products and services must fulfill their specifications to safeguard customer satisfaction and comply with regulations. The consequence of a product’s malfunctioning varies, however, depending on the application. For example, the radiation level for X-ray applications can affect health risks, whereas the strength of a coffee affects only customer dissatisfaction. The levels of confidence and conformance required for the achievement of a parameter are therefore linked to the potential consequences of a malfunction.

So how can development teams define an effective test plan that takes into account the critical risk parameters, and what sample size does this require?

**Tolerance intervals as best insight into market performance**

The test results provide the performance of a sample of products. This sample gives an insight into the tolerance interval: the range of values for a parameter for products as supplied to customers. The terminology and construction of tolerance intervals for continuous parameters are defined in ISO 16269-part 6 (Determination of statistical tolerance intervals)^{1} thus: **Statistical tolerance interval **—interval determined from a random sample in such a way that one may have a specified level of confidence that the interval covers at least a specified proportion of the sampled population.

As mentioned, the specific level of the required proportion and required confidence depends on the application and, more specifically, on the risks resulting from malfunctions where a parameter does not comply with its requirements. There is, however, no guideline stated for this. So the development team has to define the confidence requirement and the proportion requirement on the basis of their insights into the product’s use and the related risks.

**Requirements for the test plan**

Inferential statistics are applied to establish the insight into the population on the basis of a sample. In this case, we want an estimate of the range of the parameter values for the individual products to be produced.

Note that in the case of a specific pass/fail performance, we want to know the minimum percentage of products in the population that will have a good performance with a specific minimum confidence. We will focus here, however, on parameters with continuous values.

For the estimate of the tolerance interval, it’s important to consider the following:

- Is the sample representative of the population?

Is variation within the population appropriately reflected within the sample? (Consider e.g. variation between batches, cavities, production installations, etc). - What is the underlying distribution of the parameter values?

What is the assumption or physical insight for the distribution of the parameter values in the population? Can we test this assumption on the basis of the sample values? And where we are not confident about the distribution of the parameter values, what is the way forward? - Is the sample size sufficient?

The estimate of the tolerance interval is based on the sample of test results. The uncertainty in the estimates of the tolerance interval depends on the sample size.

**Tolerance interval estimations**

Once the appropriate confidence level and proportion level for the tolerance interval have been chosen, the estimate of the tolerance interval on the basis of the test result can be determined. As stated, the assumption concerning the distribution for long-term performance of the parameter is very important here. In many situations, the normal distribution will be an appropriate assumption; but it’s good to have a clear rationale as to why this is appropriate here, as deviations from normal distributions can have a major impact on the estimate of the tolerance interval. There are numerous tests available to check whether the distribution of the test values deviate from a normal distribution. We recommend the Anderson Darling test, combined with a normal probability plot of the sample data.

Where the parameter values in the population can be assumed to follow a normal distribution, we can then estimate the tolerance interval {Y_{L};Y_{U}} on the basis of the sample values, using a straightforward approximation of the exact tolerance interval as introduced by Howe^{2} (See Figure 1 for details on calculations). As shown, the resulting tolerance interval is determined by the k-factors k_{2} or k_{1} on the basis of the sample average and sample standard deviation:

For this reason, it is also referred to as the k-value statistic. As you can see, the resulting tolerance interval is dependent upon the sample size of your test results, the selected proportion level and the selected confidence level.

The exact k values for specific levels of proportion and confidence are available in the ISO standard for tolerance intervals, ISO16269-6, or can be calculated with statistical software such as Minitab.

Where the assumption of a normal distribution of the parameter values is not appropriate, there is the option to determine a ‘non-parametric tolerance interval’. However estimates of the tolerance intervals will be much wider here (see e.g. Bury^{3} for a way to determine the tolerance intervals for continuous parameters with unknown distribution).

A quantitative insight/estimate of tolerance intervals for parameter values can be established on the basis of test results. Note, however, that the opposite approach is also applicable, by asking what sample size for the test phase is adequate to have a specific level of confidence in the resulting tolerance interval and actual performance for individual products. So where we have an a priori insight, or engineering estimate of the parameter’s spread, then, on the basis of the desired confidence level and desired proportion level, we can determine the desired k value, and from that get an insight into the sample size needed for the tests.

Figure 1 calculation of approximate tolerance limits for normal distributed parameter

**References:
**1. ISO 16269-part 6 Determination of statistical tolerance intervals

- HOWE, W. G. (1969) Two-sided tolerance limits for normal populations. In
*Journal of the American Statistical Association*, 1969, vol. 64, p. 610-620.

3. Bury, K. (1999),*Statistical Distributions in Engineering*, Cambridge University Press.