The EPA2 protocol from CLSI. • Uses control material with assigned concentration (e g from external quality control) or certified reference materials. We are pleased to have a guest essay explaining the latest in Method Verification , specifically the newest version of the CLSI guideline EP15 on Method. CLSI document EPA2 describes the protocols that should be undertaken by the user to verify precision claims by a manufacturer. Precision claims by a.
|Country:||Saint Kitts and Nevis|
|Published (Last):||7 February 2012|
|PDF File Size:||14.55 Mb|
|ePub File Size:||6.97 Mb|
|Price:||Free* [*Free Regsitration Required]|
For labs clzi a quick check to insure their methods are meeting manufacturer specifications, EP15 may be the right choice. This is its fourth iteration, and although it retains much of its original approach, there were some significant changes in the A3 version.
The most significant change is the creation of a relatively simple experiment that gives reliable estimates of a measurement procedure’s a and its bias. The essentials to accomplish this were present in EP15 through all of its previous versions, but they are refined clsii combined in EPA3 to make a single experiment. Here’s a brief description of the protocol. The user should ascertain that the imprecision of the candidate measurement procedure meets the criterion for allowable imprecision before beginning the evaluation.
If the clsii procedure’s imprecision reported in publications, such as the manufacturer’s stated clssi, does not meet the criterion, the precision verification procedure described in EPA3 is not appropriate. EP15 first describes a precision verification experiment. If the user is evaluating a procedure for which there are manufacturer’s precision claims, or published precision results, that were developed using CLSI EP5, the user can verify the published precision in an experiment lasting as few as five days.
Patient samples, reference materials, proficiency testing samples, or control materials may be used as the test samples, provided there is sufficient sample material for testing each sample five times per run for five to seven runs.
Precision should be tested with two or more sample materials at different medical decision point concentrations. The experiment produces at least 25 replicates collected over at least 5 days for each sample material. The repeatability previously termed “within-run” and the eep15 previously termed “total” standard deviations are calculated by an analysis of variance technique ANOVA that properly accounts for the within-run and between-run contributions to the overall imprecision of the measurement procedure.
The user needs access to software to do the ANOVA calculations, but they are available in Excel, Minitab, Analyze-it, and other software z2 that do statistical calculations. The repeatability and within-laboratory standard deviations rp15 then compared eo15 the claimed or published standard deviations. If the calculated standard deviations are less than the published values, the user has verified the claim.
Sometimes the calculated standard deviations may exceed the published values, and yet the true standard deviations are less than the published values. For example, if the true standard deviations were actually exactly equal to their claimed counterparts, the calculated standard deviations would exceed their published counterparts fifty percent of the time in verification experiments.
To allow for this possibility, the user calculates a “verification limit” based on the published standard deviation and the size of the user’s experiment.
If the calculated standard deviation is less than the verification limit, it is not statistically significantly larger than the published standard deviation, and the user has verified the published precision. If the calculated precision exceeds the verification limit, the calculated standard deviation is statistically significantly larger than the e15 standard deviation, e1p5 the user has failed to verify the published imprecision.
The document includes tables to simplify the calculation of the verification limit. Because the precision experiment has so many replicate measurements, collected over several days, results from the precision experiment may be used to make a reliable estimate of the bias of the measurement procedure relative to the assigned target values of the sample materials used in the experiment.
The only requirement is that the assigned value must be available. The choice of material depends on the purpose of the user in estimating the bias. Two or more appropriate materials should be tested a22 the precision experiment.
If the user is interested in estimating bias relative to the peer group for ep5 testing, and wants to estimate how the measurement procedure will perform well on proficiency testing, proficiency testing materials with peer group values for the measurement procedure being evaluated are appropriate.
For epp15 relative to the quality control peer group, quality control materials with peer group values for the measurement procedure are appropriate. Typically, there is no way to estimate the uncertainty of the “assayed” values, which is needed to determine if the calculated bias is statistically significant. Internationally recognized flsi order reference materials, such as a material from the U. National Institute of Standards and Technology, or from the Joint Committee for Traceability clso Laboratory Medicine, or from similar organizations may be appropriate if the user wishes to estimate the bias relative to the assigned concentrations of such materials.
Use of these materials is important in establishing the traceability of measurement procedures. Patient samples or control materials which have been repeatedly assayed with a measurement procedure felt to be substantially equivalent to the measurement procedure being evaluated may be appropriate if the user is interested in estimating bias relative to that measurement procedure.
This could be useful, for example, if the intent of the experiment was to estimate the bias of one laboratory in a system relative to another, or to the mean of the laboratories in a system. If the sample materials are appropriate, and target concentrations are available, the user can estimate the bias between the mean concentration calculated in the precision experiment relative to the target concentration of each of the materials. The width of the verification interval depends on the uncertainty of the target value of the reference material and the standard error of the calculated mean concentration from the experiment.
Calculation of the verification interval would be complicated, but the committee simplified it greatly by providing tables for the difficult-to-calculate quantities based on the number of replicate measurements per run, the number of runs, and the uncertainty of the target value. If the mean concentration from the user’s experiment is within the verification interval, there is no statistically significant bias.
If the mean concentration from the user’s experiment is beyond the verification interval, statistically significant bias exists. The user must evaluate the estimated bias versus allowable bias.
If the estimated bias is less than allowable bias, the bias is acceptable. If the estimated bias exceeds allowable bias, it is not acceptable. Previous versions of EP15 cldi a small comparison experiment, involving 20 patient samples, which was to be used to verify a manufacturer’s claimed bias.
There were two problems with this approach.
Evaluating Assay Precision
First, users rarely have access to the measurement procedure used by the manufacturer or authors of a publication as the comparative method for the published bias.
Csli the manufacturer identifies the comparative measurement procedure only generically.
Second, most manufacturers provide only regression statistics as the results of comparison experiments, and do not provide bias claims, so the user has to calculate the bias to be expected from the regression statistics provided and has little idea of the uncertainty of this estimated bias. The EPA3 committee felt that the patient comparison experiment had little value as it was, and that users e1p5 needed to perform a patient comparison experiment should consult CLSI EP9-A3 “Measurement procedure comparison and bias estimation using patient samples.
This is valuable when the user wishes to verify precision and to estimate bias relative to a peer group or target concentration.
Evaluating Assay Precision
It may be especially useful when patient samples are difficult to obtain for a traditional comparison of methods experiment.
The EPA3 document development committee was team of experts who worked together well. Tools, Technologies and Training for Healthcare Laboratories. Verification of Precision EP15 first describes a precision verification experiment. Estimation of Bias Because the precision experiment has so many replicate measurements, collected over several days, results from the precision experiment may be used to make a reliable estimate of the bias of the measurement procedure relative to the assigned target values of the sample materials used in the experiment.
Acknowledge Committee Members The EPA3 document development committee was team of experts who worked together well.