Coronavirus (Covid-19) testing accuracy is coming to the fore as a topic of discussion and the subject of public health policy around the world.
With this conversation, we became really interested in the U.S. Food and Drug Administration approval process. If it’s FDA-approved, it must be good to go, right? Wait, not so fast.
Our post about testing accuracy from several weeks ago notes that accuracy can be as low as 70 percent for the nasal swab test, with the swab of the nose or nasal cavity being checked for the presence of the virus in someone’s respiratory tract. That is currently the most common test. That shows current infection.
Another test that is becoming more common is the serological test for antibodies: Blood is assessed for presence of antibodies that show the person has been exposed to the virus. That would show previous infection. Those tests are being discussed as a way of “opening up the economy” by allowing people who test positive to return to normal activities — on the assumption that the presence of antibodies confers immunity, something that scientists don’t universally agree upon. Yet the reliability of the antibody tests has also been called into question, notably in this New York Times article discussing a group of scientists’ work assessing their accuracy and finding it lacking.
Among the anecdotes we have heard about testing reliability:
- A patient was tested for coronavirus before entering a hospital, but results were slow in coming. On admission, the patient was tested and sent to a Covid ward on the basis of symptoms. The test came back negative, and the patient was moved out of Covid ward; then the previous test results came back 11 days later, showing positive.
- A doctor seeking to return an employee to work after her Covid illness wanted to get two negative test results, as advised by health officials. He said he got a negative test, a positive one, a negative, a positive — using seven tests in all, and never getting two serial negative tests.
- “These tests are crucial to reopening the economy, but public health experts have raised urgent concerns about their quality,” the Times article says. “The new research, completed just days ago and posted online Friday, confirmed some of those fears: Of the 14 tests, only three delivered consistently reliable results. Even the best had some flaws.”
Update May 4:“The Food and Drug Administration announced on Monday that companies selling coronavirus antibody tests must submit data proving accuracy within the next 10 days or face removal from the market,” The New York Times reported. “Since mid-March, the agency has permitted dozens of manufacturers to sell the tests without providing evidence that they are accurate — and many are wildly off the mark. …The F.D.A.’s action follows a report by more than 50 scientists, which found that only three out of 14 antibody tests gave consistently reliable results, and even the best had flaws. An evaluation by the National Institutes of Health has also found ‘a concerning number’ of commercial tests that are performing poorly, the agency said.”
How are tests approved?
In the United States, the question of accuracy focuses on the approval process for such tests with the FDA.
Generally the FDA will approve tests and devices based on a fairly limited study, experts say. The FDA itself says every product carries some risk — and sometimes those risks come to light only after the medication, device or test is on the market and is being used in a larger number of patients and for a longer period of time, with people of different health characteristics.
Then the agency collects real-world experiences after approval, in what’s called “the aftermarket,” using a database called MedWatch. Manufacturers, doctors, patients and others are able to contribute their reactions to this database.
About 95 percent of the MedWatch responses are from manufactures, while 5 percent come from doctors and patients, according to the FDA. Manufacturers are required by law to submit errors or adverse events. But for patients and doctors, it’s voluntary — and many patients don’t even know that the database is there.
In the case of coronavirus tests, an “emergency use authorization” has been granted by the FDA to some tests, but not all. These are less rigorous than the usual process. Meanwhile, more than 100 others are on the market without even the lightest of prior review.
Tests being used by the majority of testing sites for the presence of an active case of coronavirus are a polymerase chain reaction test, which is used to amplify sections of RNA in a sample. Both WHO and CDC tests use this PCR test. “It’s worth noting that PCR tests can be very labour intensive, with several stages at which errors may occur between sampling and analysis. False negatives can occur up to 30% of the time with different PCR tests, meaning they’re more useful for confirming the presence of an infection than giving a patient the all-clear,” Chloe Kent writes in Medical Device Network.
The PCR tests have been very hard to get since the beginning of the pandemic.
Separately, the serological test for the antibody is designed to find the presence of antibodies, meaning that a person has been exposed to the virus and developed an immune response, perhaps even without actually having symptoms of coronavirus. Reliability is also very much in question here.
These antibody tests are hard to find too, but there are more and more of them — we have heard of eBay purchases, and the Google search bar will lead you to sources for both tests.
“The Food and Drug Administration, criticized for slowness in authorizing tests to detect coronavirus infections, has taken a strikingly different approach to antibody tests, allowing more than 90 on the market without prior review, including some that are being marketed fraudulently and are of dubious quality, according to testing experts and the agency itself,” Laurie McGinley writes in The Washington Post.
“On Thursday, during an interview with Washington Post Live, FDA Commissioner Stephen Hahn said ‘people should be very cautious’ about tests that have not undergone the rigorous process of getting emergency use authorization from the agency. So far, only four tests have gotten such authorizations: Cellex, Chembio Diagnostic Systems, Ortho Clinical Diagnostics and Mount Sinai Laboratory.” (Since then, others have gotten emergency authorization.)
How the FDA describes its work
The FDA issued a statement in early April that tried to make clear the distinction. “In March, the FDA issued a policy to allow developers of certain serological tests to begin to market or use their tests once they have performed the appropriate evaluation to determine that their tests are accurate and reliable. This includes allowing developers to market their tests without prior FDA review if certain conditions outlined in the guidance document are met.” So the tests are being marketed with the developers issuing themselves their own approvals.
In both cases, a bad test — a false negative for the PCR test, or a false positive for the antibody test — could have life-changing consequences for the individual and society at large.
If a test for, say the presence of Vitamin D in the blood, has an error rate of several percentage points, then maybe it’s not such a big deal.
But with coronavirus testing, the increasing number of “false negative” results in the nasal swab test is leaving an unsettled aura around the idea of testing. The same is true for the new blood tests with “false positive” results.
While only a very few blood tests have received FDA approval for use, health departments, clinics, health systems and others have bought these tests to do antibody surveys, but their reliability is not certain.
“The Food and Drug Administration has allowed about 90 companies, many based in China, to sell tests that have not gotten government vetting, saying the pandemic warrants an urgent response,” Steve Eder, Megan Twohey and Apoorva Mandavilli wrote for The New York Times April 19. “But the agency has since warned that some of those businesses are making false claims about their products; health officials, like their counterparts overseas, have found others deeply flawed.”
The Times article added: “Tests of ‘frankly dubious quality’ have flooded the American market, said Scott Becker, executive director of the Association of Public Health Laboratories. Many of them, akin to home pregnancy tests, are easy to take and promise rapid results.”
A doctor’s views
Colin West, a general internist and a biostatistician at Mayo Clinic in Minnesota, is a professor of medicine, medical education and biostatistics at Mayo. He was one of several co-authors on a Mayo paper about testing, pointing to the threat of false negative results. He said:
There’s been some talk about false negative tests, we should be worried about these. But there’s a there’s perhaps a more subtle point in our paper that people have skipped over.
And that is, we need good data on the performance of these tests. And that data needs to be from scientifically rigorous studies. I think in everyone’s well intended efforts, you know, laboratories and scientists have been working night and day to develop tests for us, but the FDA actually has, in some cases, relaxed standards for approval of tests. And on the one hand, we want to get tests out there so that we can identify clusters of infection as quickly as possible. But if we relax our standards too much, and aren’t mindful that we’ve done that, then we’re going to get false negative results, false positive results, and that’s going to lead to confusion.
And so, the methodology behind testing, whether these tests are good, is something that is poorly understood. If you look up these tests, and try to get the package inserts and things like that, you find precious little information: what’s their sensitivity? What’s their specificity? What is the false negative rate? And if they do report those numbers, how did they get those numbers? And did they follow a scientifically rigorous methodology? Or is this all “within the laboratory, not in the real life situation” sort of data, which probably represents a best case scenario for test performance?
The number that we wrote about, potentially a 30% false negative rate with these tests, comes from one of the early PCR studies from China. For the United States tests, it’s been very, very difficult to get any kind of credible information about what’s the false negative rate or false positive rate for these studies. It’s not being reported.
So we’re flying blind. We are trusting that these tests are telling us what we want them to tell us. But we don’t really know that.
I’m not a microbiologist.I’m not a test developer. But I do have some savvy around these issues of test performance and reading the literature. Critical appraisal is something I do, it’s something I teach. I’ve looked at the literature and I can’t find this information. So if I can’t find it, how are our public health officials, our politicians, our leaders — where are they getting this information?
We need more of that, so that we can have a fighting chance at making informed decisions about national policies to try and get on top of the pandemic.
The speed tradeoff in the FDA process
Discussing the speed in the FDA process, West said:
It’s concerning when things get rushed like this. But I would also point out that the FDA is in a tough position because if they are seen as delaying for, you know, “what’s the threshold for the science is good enough?” — if we wait for perfection, then we have traded one problem for another. So I’m somewhat sympathetic with their decision-making — they’re going to get criticized on all sides, no matter what they do.
But I do think we have to be mindful that O.K., if we’re rushing these things through and relaxing our standards, then the tests aren’t going to be as good as they could be. And let’s just be open about that.
I think there’s a way to communicate that that isn’t terrifying. It’s more “we want you to be aware, because knowledge is power.”
We just want to make clear to people. Testing is important, but it’s not the answer. It’s part of a much larger approach. And we’re not going to get away from physical distancing, potentially masks and other personal protective measures anytime soon. The tests aren’t going to get us out of that by themselves.
A patient advocate speaks
Randi Redmond Oster, a friend who serves on an FDA panel approving pediatric use of medications, wrote an op-ed recently about the F.D.A. role in the use of hydroxychloroquine that was published widely. She wrote:
“The FDA monitors safety, efficacy and security of drugs and food in the U.S. It regulates $2.4 trillion worth of food and drugs. It has rigid policies and guidelines for drug development; typically, it takes years of testing and approvals before a new drug comes to market. … However, with the Covid-19 pandemic, we find ourselves in uncharted territory. The use of hydroxychloroquine might seem like a fast drug approval but to be clear, consumers will be testing the use of this drug on themselves for coronavirus.”
Oster is on the Pediatric Advisory Committee to the FDA, one of 12 voting members. She is the lone consumer representative.
Here’s an FDA presentation about its approval process.
Here is a whiteboard video from the Patient Safety Action network about chloroquinine and what patients need to know.
Here is a study about ineffectiveness in drugs by the FDA. On page 3 for how many reports come from doctors and patients and page 12 for how many reports deemed useful.
Medwatch is the place for adverse events to be reported: Reports can be submitted and data can also be searched on this page.
This page describes Covid-19 authorizations.
Any number of devices and medications approved by the F.D.A. have had adverse effects. Consider these:
· Breast implants, which after approval were found to be leaking and causing severe health issues, and are now sold with a box warning
· Essure birth control, approved in 2002 and removed from the market in 2018
· Ciprofloxacin, the antibiotic that is now sold with multiple warnings about side effects on mental health and “disabling and potentially permanent side effects involving tendons, muscles, joints, nerves and the central nervous system”
· Singulair for asthma and allergies, which despite its approval was found to have serious mental health effects that resulted in a black box warning just a short time ago this spring.
A doctor who says the FDA won’t let him make a test
Brian H. Shirts, M.D., is a molecular pathologist specializing in cancer genetics and an associate professor of laboratory medicine at the University of Washington.
“February was a frustrating month for my laboratory. We wanted to make tests to detect the virus that causes Covid-19. My virology colleagues had great ideas and solid testing platforms. The Food and Drug Administration told us to stop,” Brian Shirts, M.D. wrote on Stat recently. “Why? Because of a quirk in FDA regulations. Diagnostic tests are currently regulated in one of two ways, and there’s no clear rule to determine which one applies to coronavirus tests. This uncertainty is a big part of why test shortages have caused a national crisis.”
Shirts writes that “laboratory directors are licensed by their states to develop tests through a set of rules called the Clinical Laboratory Improvement Amendments (CLIA).” But “In early February, the FDA issued an Emergency Use Authorization and by doing so declared that coronavirus tests are medical devices. It assumed authority and chose to work exclusively with the Centers for Disease Control and Prevention to authorize Covid-19 testing. My laboratory, like many other CLIA-licensed labs, had already started developing coronavirus tests. But once the FDA took control, we could not offer those tests to doctors. If we did, the FDA could shut down our hospital labs.”
He places blame on The VALID Act, introduced in Congress in early March, which seeks to sort out “confusion about who regulates diagnostic testing, but it would make the situation worse.”
“Supporters of the VALID Act claim the legislation regulates tests that currently avoid oversight. That’s not true. CLIA already regulates those tests. CLIA also has an extra layer of protection: it allows a healthy marketplace where multiple competing diagnostic tests are available, “
What’s the best course forward?
For a person whose life depends on the right answer, I turned to Donna R. Cryer, founder and CEO of the Global Liver Institute, based in Washington. A transplant recipient who is an expert on immunocompromise, she has an intense professional and personal interest in getting tests and treatments right. She said:
Two wrongs don’t make a right. FDA was too slow to act at the beginning and too narrow in handling of things, so universities and well-qualified certified labs didn’t feel they were able to enter into this work and develop tests to make up for CDC lapses.
The pendulum swung too far — they threw the doors open, and any test made by anybody is available on the market. That is also not the right place to calibrate.
They need to find a place that is responsive and nimble, and proves a wider lane than usual for rapid regulatory review and approval.
She pointed out that Dr. Scott Gottlieb, former head of the FDA from 2017-2019, and Mark McClellan, head of the FDA from 2002-2004, published a paper at Duke University in mid-March titled “Advancing Treatments to Save Lives and Reduce the Risk of COVID-19,” suggesting a very different way of thinking about speed and accuracy. Cryer pointed to their work as a path forward. “The co-authors call for the FDA to establish two task forces: one focused rapid development of point of care (POC) diagnostics; the second focused on rapid development of effective therapeutics and prophylaxis. In addition, the commissioners urge the White House to accelerate steps on a nationwide COVID-19 surveillance partnership to support these efforts and help target further interventions,” Duke summarized the paper in a press release.
“Scott Gottlieb has been very insightful of his appraisal of how FDA has functioned at this time,” she said. “Look at his work with Mark McClellan. The map they developed has very good insights.”
What you can do
Oster wound up her op-ed piece with a list of things you can do. It focuses on the use of any medication for coronavirus, but it would also apply for any other medication, device or test.
“The good news is the FDA has a process to capture adverse drug events. It is the Medwatch system, an online tool that allows patients and doctors to submit their experience: https://www.accessdata.fda.gov/scripts/medwatch/index.cfm.
What you can do if prescribed medications for coronavirus or anything else:
1. Ask the doctor if using the drug is off-label
2. Ask what the side effects are of the drug when used as specified
3. Look up the label warnings
4. Determine if the severity of your condition outweighs the risks of the drug
5. Report any adverse events to the Medwatch system
I’ll add a few more, if you’re thinking of getting a coronavirus test:
-
- Ask who made the test
- Ask what is its methodology, and what are the results
- Ask to see a package insert or written material about the test
- Ask if it’s approved by the FDA.
- Check on this list of approved tests.
But remember what we told you above: Even if it’s approved by the FDA, that does not mean it’s been widely tested or is infallible.
- Ask how much it will cost. If they’re charging you, how much are they charging, and does this smell like a moneymaking scheme? I recently got an antibody test free under the New York State testing program, using a test developed by the Wadsworth Center, at the New York State Department of Health. It has an “emergency use authorization.” I wrote about it here.
- Think critically. My test was administered by the New York State Department of Health; another provider, say “Mens Health” in Texas, might not be using an authorized test. Did you find the site on Google search or through your county or state health department? This post of ours lists a number of testing sites.
- Remember that in any of these tests, errors exist. So if you get a “you’re all clear” result from a nasal swab, remember that there are a lot of false negatives, and doctors say that if you have symptoms, whether you have a negative or a positive result, you should think that you’ve got it. Also, if you have a positive result in an antibody test, remember that there could easily be false positives, and doctors don’t know if positive results for presence of the antibodies means you’re immune. So don’t act as if you’re in the clear.
