Earlier this month, shares in Novavax, a small biotechnology company, jumped 10% when its COVID-19 vaccine candidate showed promise in an early stage clinical trial.
Among the results that most excited investors were the high level of antibodies researchers had found in blood samples of those injected with the vaccine. Comparing the antibody counts—known as quantitative serum titers, or often simply “titers”—between vaccine candidates has become a parlor game of sorts for stock analysts, investors and venture capitalists seeking an early indication of which of the more than 165 vaccine candidates currently under development might prove most effective, and become the preferred inoculation for much of the world’s population.
In one bullish assessment, Evercore ISI analyst Josh Schimmer noted that Novavax’s antibody levels were “well above” what Novavax’s competitors had reported. J.P. Morgan analyst Eric Joseph wrote that “it’s not too far a stretch to conclude the [neutralizing antibody] activity of [Novavax’s vaccine candidate] looks best-in-class.”
Differences in reported neutralizing antibody titers also explains why, during the third week in July, when both Oxford University’s COVID-19 vaccine team and startup BioNTech both reported positive results from Phase 1 clinical trials of their respective vaccines, shares in AstraZeneca, which is working with Oxford, slumped while those of Pfizer, which is working with BioNTech, leapt: the titer figures were higher in the BioNTech research.
But scientists who study vaccines say there’s a major problem with such comparisons: they are, at this stage, completely invalid.
No “meaningful value”
Currently, it is impossible to make an apples-to-apples comparison of immune responses in blood tests between Covid-19 vaccine candidates. So far, each vaccine candidate has been tested using a different procedure—known as an assay— conducted in different labs. And it turns out that tests conducted this way produce wildly different results, even for the same vaccine.
“I don’t believe the neutralizing antibody numbers being reported right now have any meaningful value,” says John Moore, a professor of microbiology and immunology at Cornell University medical school who has written about the difficulties of comparing COVID-19 vaccine research results. “We don’t even know what a protective titer actually is: is a titer of 100 good or bad? We don’t know.”
Wayne Koff, the chief executive of the Human Vaccines Project, a nonprofit that is trying to speed the development of vaccines for numerous, says that most research teams working on Covid-19 vaccines are testing them on assays of their own choosing, often in their own labs. “This provides every opportunity of hyping a candidate and making it appear that one vaccine is better than another vaccine, when actually we don’t have the data to reach those conclusions,” he says.
Many research groups have been publishing comparisons of the antibody titers from the blood of people inoculated with their vaccine candidates to the antibody titers found in the blood of people who have recovered from Covid-19. The implication is that if the vaccine elicits a similar or higher antibody titer, and if people who’ve recovered from Covid-19 are immune from catching the disease again for some period of time, then the vaccine should also provide temporary immunity.
But even this analysis is flawed, Koff says. That’s because at the moment there is no standard panel of blood from recovered Covid-19 patients against which to assess the titers. The blood samples in some studies might have come from patients who were much more sick than in other studies and the level of antibody production can vary immensely between individuals depending on factors ranging from age to genetics.
A familiar problem
The problem of assay disagreement—that different tests produce different results—is well known among vaccine researchers. Moore and Koff, who both studied HIV/AIDS earlier in their careers, said that similar problems once plagued attempts to develop a vaccine for that disease. But, over the past decade, with encouragement and funding from the Gates Foundation, the field has agreed on a standard assay and designated just a few laboratories that are certified to carry out standardized testing for all vaccine candidates.
Some researchers want to replicate that system for COVID-19. Scientists at Oxford’s Jenner Institute, in publishing results of their Phase 1 human clinical trial, noted that they had tested blood samples on four different types of assay, and while the results correlated, the titers varied widely. In the paper, they called for a central lab to be set up to assess all Covid-19 vaccines.
Sarah Gilbert, one of the Oxford researchers, says a standardized assay and a central testing lab could accelerate getting multiple Covid-19 vaccines into production, something most analysts now think will be essential in order to distribute a vaccine to enough of the world’s population to end the pandemic. If one vaccine completed human safety-and-efficacy trials, and its neutralizing antibody titers were known, and a second vaccine showed the same or higher antibody titers on the same assay performed in the same lab, then this would be reasonable evidence that the second vaccine was also effective.
As a result, she says, the second vaccine might potentially be able to win regulatory approval without having to complete large scale Phase III testing. (The vaccine would still have to undergo human safety testing, of course.) “You don’t want to have to do Phase III studies for every vaccine out there,” she says.
Operation Warp Speed
The U.S. Government effort to accelerate development of an effective Covid-19 vaccine, known as Operation Warp Speed, has specified that all vaccines seeking a Food and Drug Administration license must be evaluated on the same neutralizing antibody assay in the same lab. That’s one of the reasons Operation Warp Speed’s findings will be so important to the global effort to find an effective vaccine.
John Mascola, the head of vaccine research for the National Institute of Allergies and Infectious Diseases, who has helped set up Operation Warp Speed, says that the project is currently in the process of creating and validating standardized assays. This includes designing strict procedures for how a lab should conduct the tests. He says this work should be completed “in the next few months, or sooner.” He says the lab designated by Operation Warp Speed for assessing vaccine candidates should also be named in the next few months.
He says the Operation Warp Speed is also hoping to eventually see a standardized panel of blood from recovered Covid-19 patients that can be used to compare the immune response the vaccines elicit to the immune response the actual virus creates, but that creating this panel was a lower priority and was being worked on in collaboration with the World Health Organization and European health agencies.
The issue of standardized antiboday assays is further complicated by the fact that different types of assays may be needed. True neutralizing-antibody assays, which are the gold standard, use a live SARS-CoV-2 virus, which means they have to be conducted in a specialized lab with heightened safety and security protocols. In practice, this often means a government facility. In the U.K., the neutralizing-antibody testing is being carried out at the Centre for Applied Microbiology and Research, which is housed within the Defence Science and Technology Laboratory at Porton Down, near the town of Salisbury. Porton Down is also where the U.K. Conducts much of its biological and chemical weapons research.
The protocols needed to handle a live virus, Gilbert says, makes true neutralizing-antibody assays slow to run. “It is never going to be a high throughput assay,” she says. To speed up testing and make assays available to more labs, the U.S. And other governments are also working to create a standardized “pseudovirus neutralizing assay.” This uses a harmless virus, usually a retrovirus, that is modified to produce the same external envelope of carbohydrates and proteins as the SARS-CoV-2 virus. If the antibodies a vaccine induces can neutralize this modified retrovirus, it is a reasonable guess—although not a guarantee—that they will also be able to knock out the coronavirus itself.
Blinding assays
Finally, there are what are known as “binding assays.” These simply test whether antibodies produced by a vaccine can latch on to the spike protein SARS-CoV-2 has on its surface. They do this by mixing the appropriate chemical proteins—unattached to any kind of virus—with the antibodies. Binding assays are considered less reliable indicators of a vaccine’s power than the two types of neutralizing-antibody assays, but they are simpler, cheaper and faster to perform.
Many of the same issues that bedevil comparing neutralizing antibody assays, apply equally to assays for T-cell responses. T-cells, which hunt down and neutralize infected cells and also signal the immune system to ramp up its response to a disease, are thought to be able to remember certain pathogens they’ve battled before. Many of the groups working on COVID-19 vaccines believe their inoculations will elicit a strong and long-lasting T-cell response that could confer immunity on those vaccinated, even if the antibodies prove short-lived.
But Mascola says efforts to create standardized T-cell assays are, if anything, even more fraught than creating standard antibody assays. That’s because, he says, T-cell assays require a more complex blood sample that preserves both the serum—or fluid part of blood—and the individual blood cells. These blood samples then have to be carefully frozen and unfrozen, creating a trickier protocol for standardization. Despite these difficulties, there are plans to create a standardized T-cell assay, Mascola says, but it is likely to take more time than the antibody assays.
Mascola also emphasizes that while the assay titers are important and might one day allow the short-circuiting of full Phase III clinical trials that Gilbert hopes for, the data that will determine which vaccines initially receive licenses from the FDA, and probably from most other government regulators around the globe, will be the efficacy results from large, Phase III human trials. Those trials track an inoculated group of individuals against a control group and compare how many people get COVID-19 in each.
It is only after some vaccines have proven themselves in Phase III trials that the antibody titers and T-cell numbers, performed on standardized assays, will provide a way to judge the strength of subsequent vaccine candidates against those already licensed, he says.
“There’s really no substitute for a Phase III trial,” he says.
