Changing Diagnostic Culture Calls for Point-of-care Preparedness — Multiplex Now, Open Prevalence Boundaries, and Build Community Resilience

Citation: Kost GJ (2022) Changing Diagnostic Culture Calls for Point-of-care Preparedness — Multiplex Now, Open Prevalence Boundaries, and Build Community Resilience, 21st Century Pathology, Volume 2 (5): 129

Abstract

The Coronavirus infectious disease 2019 (COVID-19) pandemic ushered in a new era of point-of-care testing now ubiquitous in homes, communities, urgent care, emergency rooms, and emergency medical services, such as ambulances. Medical professionals recognize the importance of self-testing, immediate results, and when positive, isolation to stop spread, maintain the labor force, and protect healthcare personnel. Test kits are available in local vending machines, free national distributions, and neighborhood pharmacies, as well as public health venues. Public health professionals transitioned from reference laboratory testing to widespread community diagnosis that mitigates adverse outcomes. The question arises — how should we prepare for the next pandemic? Even now, seasonal surges in influenza obfuscate the differential diagnosis of COVID-19. This article compares the performance of multiplex COVID-19/ Influenza A/B molecular diagnostics to that of home LAMP tests with FDA Emergency Use Authorizations. It recommends multiplex testing during influenza surges. High sensitivity assays can rule out COVID-19 infection in settings of increasing prevalence when symptoms and signs typical of influenza appear, especially in the young and elderly. Enabling widespread access to multiplex testing will require clever technological innovations to assure that self-diagnosis is accessible and affordable for responsible rapid decision making, triage of school children, and protection of the elderly. A primary benefit will be community resilience when faced with new outbreaks and global threats.

Keywords:

Coronavirus infectious disease 2019; Molecular diagnostics; Multiplex, point-of-care, home, and self-testing; False omission rate; Prevalence boundaries; Performance expectations; Infectious threats; New pandemic; Community resilience.

Running Title: Changing Diagnostic Culture

Background

In a recent issue of this journal, Lake D, et al. (2022) [1] suggests, ‘A rapid test that measures levels of neutralizing antibodies could provide a probability of infection that would allow immunocompromised individuals and their healthcare providers to make informed decisions about their healthcare and lifestyle, but only if the test reflects the viral sub-variant(s) in circulation at the time an individual takes the rapid test.’ They point out the importance of distinguishing infection from disease and further, ‘we have entered the age of communicable diseases in which individuals must protect themselves.’

Hence, it is increasingly important to distinguish Influenza A/B (Flu A/B) and respiratory syncytial virus (RSV) infections from COVID-19, which carries a higher risk of death currently averaging about 3-4 times that of Flu A/B in the elderly and immunocompromised. Development of technology that makes multiplex testing readily accessible will create high value in anticipation of the next pandemic, whether from a more lethal subvariant of COVID-19 or a newly emerging virus. The goals of this article are to assess multiplex assays, set performance expectations, and recommend paths forward to enhance community resilience.

Methods

For mathematical, visual logistic, and software approaches used to compare diagnostic test performance, please see Kost [2]. For the set of Bayesian equations underlying the math and graphics presented here, see Table S5 in the Supplement to [2] found via this link: https://www.mdpi.com/2075-4418/12/5/1216.

Briefly, predictive value geometric mean-squared (PV GM2) graphs compare the performance of different diagnostic tests over the entire range of disease prevalence. These ‘visual logistics’ are intended for pattern recognition, not point comparisons. The false omission rate, RFO, is equal to 1-NPV, where NPV is the negative predictive value, which also is a continuous function of prevalence.

The prevalence boundary (PB) is the point where the RFO exceeds a desired tolerance limit for missed diagnoses, set at 5% in this article. That means when the prevalence exceeds the PB, more than 1 in 20 diagnoses will be missed. One can adjust the RFO threshold to meet clinical objectives in specific emergency, community, and home settings [2].

The equation for PB is, PB = {y(1-t)/[(1-x) - (1-t)(1-x-y)]} = [yRFO]/[(1-x) - RFO(1-x-y)], where x is the sensitivity (or PPA), y is the specificity (or NPA), t is the NPV, and RFO is the tolerated rate of false omissions. Multiplex tests collated here were located by using search terms in the FDA EUA listing for ‘In Vitro Diagnostics EUAs-Molecular diagnostic tests for SARS-CoV-2’ [3]. ‘Flu’, ‘Influenza’ and ‘LAMP’ (loop-mediated isothermal amplification) were inserted into the FDA EUA website search box to produce short lists.

Tabulations are representative of manufacturers that feature these terms in their commercial titles, rather than comprehensive in that some FDA authorized COVID-19 tests may reside on large mainframe instruments capable of detecting numerous respiratory viruses but incapable of being used at or near sites of care. No human subjects were involved in this research.

Results

Table 1 compares diagnostic performance metrics for multiplex COVID-19/Influenza A/B molecular assays that received Emergency Use Authorizations (EUAs) from the United States Food and Drug Administration (FDA). Some include detection of RSV as noted in a right column. The metrics comprise PPA, positive percent agreement to a reference method recognized by the FDA (analogous to sensitivity in a clinical evaluation), and NPA, negative percent agreement (analogous to specificity).

PPA and NPA pairs assign assays to performance tiers defined by Kost [2] as follows: Tier 1-PPA 90%, NPA 95%; Tier 2-PPA 95%, NPA 97.5%; and Tier 3-PPA 100%, NPA≥99%. The tiers form a consistent set of progressive prevalence boundaries (PBs) for a risk level (RFO) of 5%. The PB for Tier 1 is 33.3% and for Tier 2, approximately 50%. A Tier 3 test has a PPA of 100%. It produces no false negatives (FN=0). Thus, RFO is zero since 1-NPV=1-[TN/(TN + FN)]=1-TN/TN=0.

The sum of tiers generates a score from 0 to 9 for each multiplex assay (Table 1). An additional point is assigned if the PPA for COVID-19 is 100%, meaning the assay can potentially rule out SARS-CoV-2 infection in the presence of confounding Flu A/B infections. The statistical summary at the bottom of (Table 1) lists low, median, and high-performance metrics used to graph multiplex COVID-19 assay performance.

Table 1: Molecular multiplex COVID-19 and Influenza A/B tests, tiers, and scores-based on manufacturer claims in FDA emergency use authorizations.

Organization EUA Date (11 EUAs)	COVID-19		Influenza A		Influenza B		N for Covid Flu A Flu B	Observations Cautions RSV Notes	Score (∑ Tiersa) Notes
	PPA % (CI)	NPA % (CI)	PPA % (CI)	NPA % (CI)	PPA % (CI)	NPA % (CI)
Applied BioCode 8/24/22	96.7 (83.3-99.4)	98.8 (93.6-99.8)	100 (85.7-100.0)	98.9 (94.0-99.8)	100 (72.2-100.0)	99.0 (94.8-99.8)	115 115 115	Only 10 Flu B positive samples. RSV also detected.	easyMAG extraction system.
9	Tier 2		Tier 2		Tier 3				7
Roche Mol Sys Liat 8/11/22	95.2 (88.3-98.1)	99.6 (99.0-99.9)	98.3 (95.1-99.4)	96.0 (94.7-97.0)	95.2 (84.2-98.7)	99.4 (98.8-99.7)	935 1350 1350		Prospective Covid NPS specimens.
9	Tier 2		Tier 1		Tier 2				5
Exact Sci Lab 8/2/22	95.56 (85.17-98.77)	100 (90.11-100)	94.44 (81.86-98.46)	100 (95.25-100)	96.67 (84.68-99.46)	100 (95.42-100)	80 113 113		Home collection available.
9	Tier 2		Tier 1		Tier 2				5
Opti Medical Systems 4/21/22	100 (90.82-100)	100 (90.11-100)	96.7 (83.33-100)	100 (90.11-100)	97.8 (88.43-99.61)	99.42 (90.11-100)	73 65 80	Caution: Small number of specimens.	QuantStudio 5 384-well format.
9	Tier 3 + 1a		Tier 2		Tier 2				8
Holologic 3/23/22	96.1 (86.6-98.9)	99.6 (97.8-99.9)	100 (92.6-100)	99.2 (97.1-99.8)	100 (92.7-100)	100 (98.5-100)	299 299 299
9	Tier 2		Tier 3		Tier 3				8
Thermo Fisher Sci 3/4/22	96.1 (86.5-99.5)	100 (93.9-100.0)	94.6 (85.1-98.9)	99.0 (94.8-100.0)	91.7 (77.5-98.2)	96.8 (91.9-99.1)	110 160 160	Flu A Quant Studio 5 comparator.
9	Tier 2		Tier 1		Tier 1				4
Roche Mol Sys cobas 2/14/22	96.4 (87.7-99.0)	98.0 (95.6-99.1)	100.0 (94.0-100.0)	99.6 (98.0-99.9)	100.0 (90.6-100.0)	99.7 (98.2-99.9)	348 344 344	Archived NPS samples used.	Compared to Roche Liat data.
9	Tier 1		Tier 3		Tier 3				7
Cepheid Xpert Xpress CoV-2/Flu/ RSV plus 10/20/22	100.0 (94.5-100.0)	100.0 (98.2-100.0)	100.0 (93.0-100)	100.0 (98.2-100.0)	100.0 (92.3-100.0)	100.0 (98.3-100.0)	213 213 218	Dx or Infinity Systems. RSV also detected. Concordance, see Johnson-14 & Sluimer-42 in JCVP 2021.	Claims must be proven in independent evaluations of sensitivity and specificity in community populations.
9	Tier 3 + 1a		Tier 3		Tier 3				10
CDC 8/5/22	100 (CI NR)	100 (CI NR)	100 (CI NR)	100 (CI NR)	100 (CI NR)	100 (CI NR)	74 71 71	Extreme caution: Smallest sample sizes. No Cis.	Claims must be proven in independent community evaluations.
9	Tier 3 +1a		Tier 3		Tier 3				10
NeuroMoDx Molecular 4/23/22	100 (86.7-100)	98.0 (87.9-99.9)	100 (91.1-100)	100 (91.1-100)	96.7 (80.9-99.8)	98.0 (88.0-99.9)	82 100 80	RSV also detected.	Direct workflow.
9	Tier 2		Tier 3		Tier 2				7
Cepheid POC System 1/27/21	97.9 (88.9-99.6)	100.0 (98.1-100.0)	100.0 (92.6-100.0)	100.0 (98.0-100.0)	100.0 (92.3-100.0)	99.0 (96.3-99.7)	240 240 240	RSV also detected.
9	Tier 2		Tier 3		Tier 3				8
Statistical Summary
Low	95.2	98.0	94.44	96.0	91.7	98.0
Min LB CI	83.3	87.9	81.16	90.11	72.2	88.0
High	100.0	100.0	100.0	100.0	100.0	100.0
Max UB CI	100.0	100.0	100.0	100.0	100.0	100.0
Median [N = 11]	96.7	100.0	100.0	100.0	100.0	99.42
9	Tier 2		Tier 3		Tier 3

Abbreviations: Biotech: Biotechnology; CDC: Centers for Disease Control and Prevention; CI: Confidence interval; CLIA: Clinical Laboratory Improvement Act; Covid or COVID-19: Coronavirus infectious disease 2019; Dx: Diagnostics; EUA: Emergency Use Authorization; Flu A B: Influenza A, B; FN: False negative; JCVP: Journal of Clinical Virology Plus; LB: Lower bound; Max: Maximum; Min: minimum; Mol: Molecular; N: Number of samples; NPA: Negative percent agreement; NPS: Nasopharyngeal specimens; NR: Not reported; POC: Point-of-care; PPA: Positive percent agreement; SARS-CoV-2: Severe acute respiratory syndrome-Coronavirus-2; Sci: Sciences; Sys: Systems; TP: True positive; and UB: Upper bound.

Footnote, derivation of sum score from performance tiers. a) The three tiers comprise: Tier 1 - PPA 90%, NPA 95%; Tier 2 - PPA 95%, NPA 97.5%; and Tier 3 - PPA 100%, NPA ≥99%. Multiplex tests simultaneously detect COVID-19, Influenza A, and Influenza B. A multiplex assay earns one additional point if the COVID-19 claim is Tier 3, which requires a PPA [TP/ (TP + FN)] of 100%. This suggests that no false negatives will be observed in prospective clinical validations. If so, the assay will add value by ruling out SARS-CoV-2 infection in settings of high COVID-19 prevalence when patients present with Influenza A or B. The assay also will generate very few false positives in settings of low prevalence because NPA [TN/(TN + FP)] is ≥99%.

Figure 1: Performance patterns for COVID-19 testing in multiplex formats and for LAMP home self-testing.

 

The black horizontal line in Figure 1 represents a risk level of 1 in 20 missed diagnoses. PBs increase from left to right with median LAMP performance generating a PB of 38.4%. A PB less than that of Tier 2 (50.6%) will incur high risk of missing COVID-19 diagnoses as prevalence increases. Both the LAMP home test median and the most poorly performing multiplex COVID-19 test generate risk above the tolerance level at moderate prevalence.

However, in general COVID-19 assays in multiplex formats perform well and exceed the Tier 2 threshold (PPA 95%, NPA 97.5%). Note that the curves reflect manufacturer EUA claims, which typically become challenged in clinical studies. That is, performance falls off substantially when tests are evaluated independently in real-world medical settings with large diverse populations. Two concordance studies are listed in (Table 1), and two LAMP clinical evaluations in (Table 2).

Table 2: COVID-19 molecular tests with FDA emergency use authorization for home self-testing.

Molecular Diagnostics, Details	Tier, Sample Size, and Cost	Company, Product, EUA LOA Date [Earliest Date]	Company PPA (%) Claim [CI, D]	Company NPA (%) Claim [CI, D]	Specimen Type, Age, and Time Interval/ Protocol for Specimen Collection [plus notes]
	Tier 1 (2)
RT-LAMP and lateral flow strip. ORF1ab region of the SARS-CoV-2 genome.	N=112 Test & processing hub $75.	Detect COVID-19 Molecular Non-prescription Home Test 1/12/22 [10/28/21]	90.9 [76.4-96.9, D=20.5]	97.5 [91.2-99.3, D=8.1]	AN swab. Self-collected ³ 14 or adult assisted ³ 2 years. Without symptoms performed twice > 24, < 48 hours between tests. Apparent FPs due to subject misinterpretation; app modified to reduce this error. Testing time 55-65 minutes.
RT-LAMP. Non-overlapping regions of the N gene.	N=404 Overall N=101 (1) N=303 (2) Single-use test kit $75.	Lucira CHECK-IT COVID-19 Test Kit 4/9/21 [4/9/21]	91.7 [85.6-95.8, D=10.2] (1) 94.1 [85.5-98.4] (2) 90.1 [81.5-95.6]	98.2 [95.8-99.4, D=3.6] (1) 98.0 [89.4-99.9] (2) 98.2 [95.5-99.5]	AN swab. Symptomatic (1) and asymptomatic (2). PPAs exceed Tier 1 threshold of 90%. NPAs meet the Tier 2 threshold of 97.5%. Testing time 30 minutes. A clinical evaluation of 190 patients showed sensitivity of 91.1% (S 93.06%) and specificity of 100% (Zahavi et al. Diagnostics, 2022 12:1877).
	Tier 2 (1)
“Isothermal nucleic acid amplification test.” Nucleocapsid N region of the SARS-CoV-2 virus.	N=271 Overall N=138 (1) N=133 (2) Test $61.75 to $65 each. Reader $249.	Cue Health COVID-19 Test for Home and OTC Use. 2/9/22 [3/5/21]	97.4 [86.5-99.5, D=13.0] (1) 96.4 [82.3-99.4] (2) 100 [72.2-100]	99.1 [96.9-99.8, D=2.9] (1) 98.2 [93.6-99.5] (2) 100 [97.0-100]	AN swab. Self-collected adult or assisted ³2 years. Symptomatic (1) and asymptomatic (2). Testing time 20 minutes. A clinical evaluation of 292 outpatients in a community drive through showed sensitivity of 91.7% and specificity of 98.4% for S and A (with recent exposure) adult subjects (Donato et al. Diagn. Microbiol. Infect. Dis., 2021 100:115307).
Statistics
Median, range [low-high]	N=271 [112-404]		91.7 [90.9-97.4, DM=6.5]	98.2 [97.5-99.1, DM=1.6]	Median and mean performance are both Tier 1.
Mean [SD]	262.3 [146.2]		93.3 [3.54]	[0.80]

Abbreviations: Δ: magnitude in % of the 95% CI, i.e. high minus low limits; ΔM: Span of the median range; A: Asymptomatic; AN: Anterior nares; CI: 95% confidence interval with upper and lower bounds in percent; COVID-19: Coronavirus disease 2019; EUA: Emergency Use Authorization; FDA: Food and Drug Administration (USA); RT-LAMP: Reverse transcription loop-mediated isothermal amplification; LOA: Letter of authorization; NA: Not applicable; NCP: Nucleocapsid protein; NPA: Negative percent agreement; O: Overall; OTC: Over the counter; PPA: Positive percent agreement; PPV: Positive predictive value; and S: Symptomatic.
Notes: a) Tier sensitivity/specificity (%) comprise: 1) 90/ 95; 2) 95/ 97.5; and 3) 100/ 99. b) Home collection EUAs are not listed.

Discussion

What will happen in the post-EUA era, if and when it ends? The FDA should increase performance expectations for authorization of new COVID-19 tests, including those in multiplex formats, and give current holders of EUAs an opportunity to improve assay performance. Tier 2 is attainable in EUAs, even for COVID-19 rapid antigen tests (Figure S1 in Kost [2]). With PCR-based assays, high levels of individual viruses can impair the simultaneous detection of others in a multiplex format [4]. Novel biosensors may represent a potential future solution.

Independent multicenter performance evaluations can enhance industry accountability by using diverse populations and larger sample sizes. For example, a European multicenter evaluation of the Cepheid PCR-based multiplex test at four medical institutes with a total of 295 nasopharyngeal samples revealed PPA/NPA pairs of 97.2%/100%, 95.3%/100%, and 95.7%/100% for COVID-19, Influenza A, and Influenza B assays, respectively [5], which differ from those in (Table 1).

Influenza testing can inform decision making for home care, individuals living with others at high risk of complications, and when prescribing antivirals. Confirmation of influenza virus infection generally is not required before starting antivirals. Collection of high-quality specimens and careful handling during viral shedding early in the course of illness improve test yield. Positive results can aid infection control practices, especially prevention of nosocomial spread among hospitalized patients, for whom molecular testing is recommended. Coinfection with SARS-CoV-2 may appear both in the hospital and community settings.

Conclusion

Therefore, technology should focus on improving clinical sensitivity, especially for COVID-19, so that negative results become more reliable with less uncertainty under conditions of moderate to high prevalence in different emergency, community, and home settings. Then, people will not spread disease unknowingly. Unfortunately, objective independent evaluations of community and home self-testing are not available. Even if one accepts the risk of missing 1 in 10 diagnoses, the LAMP home self-test median PB would be only 56.8%, a prevalence typical of Omicron infection.

The pandemic motivated major improvements in the accessibility of testing in emergency rooms and communities [2]. For example, Figure 2 shows a vending machine used to distribute COVID-19 tests in a small university town. However, technological innovations have yet to bring multiplex testing to homes in highly reliable, user friendly, and inexpensive molecular diagnostic formats. While LAMP is available for self-testing of COVID-19, a combination COVID-19/Flu A/B LAMP assay has yet to appear, possibly because LAMP is difficult to multiplex.

Figure 2: Vending machine distribution of COVID-19 rapid antigen test kits.
Vending machines are placed outside libraries and public health in Yolo County, the first to do so in California. One punches a number and recovers the free rapid antigen test kit below. Access to self-testing is excellent. Instructions are in the package insert. The approach could be improved by linking educational video media. No personal information is required. The case is refrigerated. One could envision ATM-like multiplex testing stations for specimen submission in these locations.

 

The COVID-19 pandemic brought significant changes in diagnostic culture. Public health now embraces POCT. Laypersons display competence in self-testing. Home testing performs best in symptomatic individuals [6]. Free choice generates self-determination. Family members protect each other and their elders by knowing who is infected and when to isolate. We must remember this important medical, social, and cultural experience.

Importantly, new National Institutes of Health funding supports these social and diagnostic advances [7]. While primarily targeting accessibility, elimination of serial testing [8], POC formats, and ease of use, the next generation COVID-19 testing should deliver clever multiplex assays capable of identifying serious infections at points of need when new global threats arise in our communities.

Author Contribution

GJK contributed all aspects.

Conflict of Interest Statement

No potential conflict of interest relevant to this article was reported.

Acknowledgment

The author is grateful to have received a Fulbright Scholar Award 2020-2022, which supports analysis of COVID-19 diagnostics and strategic POCT field research in ASEAN Member States with the goal of improving standards of crisis care. Figures and tables are provided courtesy and permission of Knowledge Optimization, Davis, California.