Discordant IVD Results: Preserve the Original Comparison Before Resolution

Keep the original comparison table and report the follow-up results separately. If your new in vitro diagnostic (IVD) and its comparator disagree on 17 specimens, sequencing those 17 can provide additional observations. It does not justify replacing the original comparator calls and reporting the improved agreement as primary performance.

FDA’s March 13, 2007 statistical guidance identifies that relabeling practice as discrepant resolution. Section 6, item 3 advises against using outcomes changed by discrepant resolution to estimate sensitivity, specificity, or agreement. Section 7.3 recommends retaining the original two-by-two (2x2) table and presenting resolver findings separately, using Table 8 as an example. FDA statistical guidance, Sections 6 and 7.3.

Which comparison are you actually reporting?

Start by deciding whether the benchmark is a reference standard or a non-reference comparator. That choice determines what the numbers mean. A reference standard is the best available method for establishing whether the target condition is present. A comparator can be useful without establishing that condition reliably enough to serve as a reference standard. Calling a method “established” or “FDA-cleared” does not, by itself, settle that distinction for your intended use.

FDA recommends sensitivity and specificity for comparison with an appropriate reference standard. For a non-reference comparator, report positive percent agreement (PPA) and negative percent agreement (NPA), together with the original table, a description of the comparator and how it was performed, and confidence intervals. Two agreeing tests can both be wrong; agreement does not establish clinical correctness. FDA statistical guidance, Sections 3, 4.1 and 7.2.

With the comparator defining the denominators, PPA is a/(a+c) and NPA is d/(b+d). State that direction explicitly: agreement calculated with candidate-positive specimens as the denominator answers a different question. The original comparator column should remain identifiable in the specimen record even after additional tests have been performed.

For the submission team, this is a data-accounting decision before it is a writing decision. A report can accurately describe an unexpected resolver result while still preserving the original comparison. Keep the candidate call, original comparator call, and follow-up call in distinct fields so a reviewer can reproduce each table without guessing which result replaced another.

Why relabeling improves the numbers without validating them

Selective resolution examines the disagreements in cells b and c while leaving concordant cells a and d unverified. In the relabeling procedure considered here, a resolver-positive result moves a candidate-positive/comparator-negative specimen from b to a. A resolver-negative result moves a candidate-negative/comparator-positive specimen from c to d. Resolver findings that favor the original comparator leave the disagreement in place.

That bookkeeping can only improve or leave unchanged the agreement calculated from the original table, provided the denominators remain defined. If x specimens move from b to a and y move from c to d, the revised fractions become (a+x)/(a+x+c−y) and (d+y)/(b−x+d+y). No initially concordant specimen becomes discordant under this procedure. The apparent improvement therefore has a built-in favorable direction, irrespective of whether the resolver establishes truth.

Even an uninformative coin-flip resolver can produce favorable reclassification by chance. FDA uses this point to show why better-looking agreement after resolution is not evidence of better diagnostic performance. Resolving a specimen may be useful for investigation; the defect arises when selectively obtained information is used to rewrite the performance comparison. FDA statistical guidance, Section 7.3.

Miller’s 1998 algebraic analysis demonstrates overestimation under discrepant analysis even when the resolver is a perfect reference standard. Hadgu’s 1999 critique likewise challenges using the new test to help determine the condition against which that same test is evaluated. These papers support the methodological concern; their abstracts do not establish present-day submission procedures. Miller, 1998. See also Hadgu, 1999.

Synthetic example: preserve the same 500 specimens

Assume one specimen per subject and no missing, invalid, or equivocal results in this simplified example. The candidate and non-reference comparator have paired results on 500 specimens: 55 concordant positives, 428 concordant negatives, 12 candidate-positive/comparator-negative specimens, and five candidate-negative/comparator-positive specimens. The original comparison is:

Accounting A: original comparison — synthetic counts

These original counts give PPA of 55/60 = 91.67% and NPA of 428/440 = 97.27%. Two-sided 95% Wilson score intervals are 81.9%–96.4% for PPA and 95.3%–98.4% for NPA. Overall agreement is 483/500 = 96.60%, with a Wilson interval of 94.6%–97.9%. These intervals describe binomial sampling uncertainty conditional on this constructed comparison; they do not correct comparator error or selective-verification bias.

The difference between the two denominators matters operationally. PPA has only 60 comparator-positive observations, even though the study total is 500. Its interval is correspondingly wider. A high overall agreement percentage, driven here by the numerous concordant negatives, should not obscure the uncertainty in positive agreement.

Follow-up ledger — 17 synthetic discordant specimens

Now suppose the team performs a separate resolver test on the 17 disagreements. The following invented identifiers make each follow-up result traceable to its original pair. They are illustrative record keys, not patient identifiers. A resolver result is an additional observation; it is not automatically a final diagnosis.

Ten of the 12 specimens in cell b are resolver-positive, and two are resolver-negative. In cell c, two are resolver-positive and three are resolver-negative. Thus 13/17 resolver calls agree with the candidate call (76.5%) in this selected discordant subset. That percentage is descriptive agreement within the subset; it is not sensitivity, specificity, or evidence that 76.5% of all candidate results are correct.

Table 8-style summary — synthetic counts

Including the untested concordant groups makes the limit visible: resolver status is unavailable for 483 specimens. “Not tested” must not become “negative,” “confirmed,” or “no error.” This summary follows the reporting idea in FDA’s Table 8, while the specimen ledger above provides a practical traceability extension. The original 2x2 table remains unchanged.

Accounting B: inappropriate relabeling — synthetic counts

If the team overwrites the comparator calls, the ten resolver-positive specimens from b move to a, and the three resolver-negative specimens from c move to d. The candidate results and total number of specimens do not change. The supposedly corrected comparison becomes:

The increases use unrounded fractions before rounding to two decimal places. They are percentage-point differences, not relative percentage increases. No assay was improved, no additional subject entered the study, and no reference-standard result was obtained for the 483 initially concordant specimens. The exercise shows improvement relative to the original agreement calculation; the true diagnostic accuracy remains unknown.

The unverified concordant specimens could contain errors, but this example supplies no evidence about whether they do. The selective design never checks those specimens, so it cannot establish the absence of concordant errors. The two accountings demonstrate how relabeling changes the reported numbers; they do not represent interchangeable ways to report primary performance.

What would support an adjusted accuracy estimate?

FDA distinguishes discrepant-only relabeling from properly designed partial-verification studies. Section 6 says unbiased sensitivity and specificity estimation requires a reference-standard resolver and resolution of at least a subset of concordant subjects. Section 4.1 describes designs using the reference standard on a random subset of all subjects, or on all disagreements plus a random sample of agreements, with appropriate adjusted estimates and variances. FDA statistical guidance, Sections 4.1 and 6.

Simply adding a few convenient concordant specimens is not a complete statistical design. Define the sampling rule, how selected subjects represent each group, and how estimation will account for verification. The simple unadjusted 2x2 sensitivity and specificity formulas do not automatically become valid for a partially verified cohort. A statistician should specify the method and precision assessment before the team treats the resolver study as an accuracy study.

For an already completed study, document what was actually planned and performed. If follow-up was decided after looking at disagreements, describe it as such rather than retroactively presenting it as a prospective comparator algorithm. Preserve the original comparison, identify the exploratory purpose, and distinguish any proposed new study or adjusted analysis from the descriptive follow-up already available.

Designing a comparator before discordance occurs

A composite comparator is a different design choice. FDA’s statistical guidance allows a reference standard comprising multiple methods, with the combining algorithm part of that standard. Section 6, item 4 warns against comparator algorithms that use the candidate test’s result. Writing a rule in advance does not remove incorporation bias if that rule still uses the new test to determine its own benchmark. FDA statistical guidance, Sections 3 and 6.

A device-specific example appears in FDA’s October 9, 2009 respiratory viral panel (RVP) multiplex nucleic acid assay special-controls guidance. Section 6.D.ii allows a predetermined multi-test comparator for viruses difficult to culture, including a separate well-characterized amplification method targeting different genomic regions, followed by bidirectional sequencing of at least 200 base pairs of acceptable quality matching a reference or consensus sequence. FDA RVP guidance, Section 6.D.ii.

Those technical details belong to that RVP guidance. They are not universal resolver requirements for all IVDs, nor do they turn selective sequencing of disagreements into a reference standard for the entire cohort. For another assay class, determine the appropriate benchmark and applicable device-specific recommendations rather than importing the 200-base-pair example into the protocol.

Within the same RVP scope, Section 6.D.i discusses supplementing rare-virus prospective specimens with pre-selected banked specimens and recommends presenting those results separately as PPA/NPA. Section 6.D.iv recommends per-virus and subtype sensitivity/specificity with 95% confidence intervals, plus reporting of co-infections. These population and presentation distinctions remain relevant even when the comparator algorithm is well designed. FDA RVP guidance, Sections 6.D.i and 6.D.iv.

Keep invalid-result retests distinct from resolver testing

A valid disagreement is different from an invalid or equivocal assay result. In its RVP guidance, FDA recommends testing according to the assay’s instructions for use (IFU). Where the IFU calls for repeating initial invalid or equivocal results, the final results enter the analysis. The guidance also recommends reporting retest percentages and final invalid/equivocal percentages by virus and overall. FDA RVP guidance, Section 6.D.iv.

For the study ledger, keep the reason for repeat testing explicit: an IFU-directed repeat following an invalid result, a defined equivocal-result pathway, or exploratory follow-up of two valid but disagreeing results. Keep both the initial and final outputs available. A resolver call should not silently replace a valid original comparator result simply because the team prefers the follow-up interpretation.

CLSI’s March 13, 2023 announcement describes EP12, third edition, as guidance for qualitative binary examinations and performance evaluation by manufacturers and laboratories, including local verification. It provides relevant evaluation context. The public announcement does not verify a particular EP12 section, create a binding submission rule, or establish a universal discordant-resolution protocol. CLSI EP12 third-edition announcement.

Where the evidence is more nuanced

Improvement relative to the original comparator estimate is not the same as bias relative to truth. Green, Black and Johnson modeled Chlamydia test evaluations with prevalence of 2%–10% and culture sensitivity of 60%–85%. In that model, discrepant-analysis specificity bias ranged from −0.63 to +0.43 percentage points, and sensitivity estimates depended strongly on whether culture specificity was exactly 100%. Offsetting biases could make an estimate closer to truth without validating the procedure. Green and colleagues, 1998.

McAdam’s 2000 commentary discusses the effects of prevalence and dependence between the candidate and resolver. It also recommends disclosing results before and after discrepant analysis when the method is used. That transparency advice is not FDA authorization to report relabeled agreement as primary performance. The 2007 FDA reporting recommendation remains the relevant source for the separate-table approach described here. McAdam, 2000.

For the manufacturer, the useful distinction is between finding an explanation and estimating performance. A sequencing result may help explain a disagreement, but selectively investigating disagreements cannot establish how often both original methods agreed incorrectly. Neither a favorable synthetic example nor a small modeled bias in one setting supplies an acceptance threshold for another device.

Map the reporting decision to its source

Prepare the discordant-specimen handoff

The following is a practical internal handoff, not an assertion that FDA mandates this exact file layout. Give the statistician, clinical writer, and RA/QA reviewer a common set of records so the performance section can be reproduced from the original observations.

• Protocol and analysis plan: identify the original comparator, its algorithm, planned follow-up, timing of decisions, analysis population, exclusions, and handling of unresolved results. Distinguish planned analyses from exploratory work.

• Original paired-result ledger: retain one record per analysis unit, with candidate and original comparator calls in separate fields. Preserve links to raw test records and reconcile each original 2x2 cell to the included records.

• Follow-up ledger: add the resolver call, method, testing reason, and availability status without overwriting either original call. Record which specimens were selected and which were not tested.

• Metrics worksheet: reproduce PPA/NPA or reference-standard accuracy measures from the intended analysis, with denominators and confidence-interval methods stated. Round percentages after computation and distinguish percentage points from relative changes.

• Descriptive resolver exhibit: display original result groups and follow-up findings, including unavailable resolver outcomes. Explain the selection process and limit interpretation to what the verified observations support.

• Retest reconciliation: keep IFU-directed invalid/equivocal repeats separate from valid disagreements. Reconcile initial results, repeat attempts, final outcomes, and exclusions so neither pathway silently drops difficult specimens.

• Report and labeling reconciliation: check that each presentation uses the same defined analysis population and original comparison. Describe comparator limitations and avoid presenting agreement as established diagnostic accuracy.

If you discover that draft tables have already been relabeled, regenerate the original comparison from retained calls, then build the follow-up exhibit independently. Reconcile the number of specimens, each original discordant group, and every resolver result between those displays. If the original calls cannot be recovered, an attractive revised percentage cannot repair that evidence gap.

For broader planning, connect this handoff to the statistical analysis plan and, where relevant, the IVDR performance-evaluation file. The accounting principle is useful across teams; FDA guidance alone does not establish EU legal duties or demonstrate what a particular notified body will accept.