Adumbrations of FDA

Scott Alexander's Adumbrations Of Aducanumab is a great review of FDA snafus -- with deeper lessons about regulation in general. Yes the outcome is dumb, but incentives are to blame. That's important to understand if we are ever to fix this mess. 

Scott has some great ideas for fixing the FDA's incentives. The one I like best is to reduce its power. FDA approval currently means that insurance companies and the government must pay for drugs. Break that link. The FDA now either decides safe&effective vs. not-yet-proven, and makes taking any not-yet-proven drug illegal. Reduce the FDA to simply providing information about what's known about drugs. Finally, give the FDA budgetary rewards for approving drugs. Bemoaning regulatory idiocy is fun but gets us nowhere. Anything persistently busted is not the result of stupidity, it is the result of bad incentives. 

FDA, CDC and Covid

The story of the FDA in covid is a good place to start. It's well known by now, but we are now in the era of forgetting, and it is to nobody's interest to keep this memory alive. 

The countries that got through COVID the best (eg South Korea and Taiwan) controlled it through test-and-trace. This allowed them to scrape by with minimal lockdown and almost no deaths. But it only worked because they started testing and tracing really quickly - almost the moment they learned that the coronavirus existed. Could the US have done equally well?

I think yes. A bunch of laboratories, universities, and health care groups came up with COVID tests before the virus was even in the US, and were 100% ready to deploy them. 

As with vaccines, which took a weekend to create, the state of medical science is such that really there is no reason to have pandemics any more. Public policy? Well, that's stuck in the 1700s.  

But when the US declared that the coronavirus was a “public health emergency”, the FDA announced that the emergency was so grave that they were banning all coronavirus testing, so that nobody could take advantage of the emergency to peddle shoddy tests. Perhaps you might feel like this is exactly the opposite of what you should do during an emergency? This is a sure sign that you will never work for the FDA.

Repeating earlier comments, there is a big difference between using tests for diagnosing very sick patients, and using tests to isolate suspected carriers and stop the spread of a disease. The latter need to be fast, deployed fast but not accurate. The FDA doesn't even have a conceptual framework to make this distinction.  

The FDA supposedly had some plan in place to get non-shoddy coronavirus tests. For a while, this plan was “send your samples to the CDC in Atlanta, we’ll allow it if and only if they do it directly in their headquarters”. But the CDC headquarters wasn’t set up for large-scale testing, and the turnaround time to send samples to Atlanta meant that people had days to go around spreading the virus before results got back. After this proved inadequate, the FDA allowed various other things. They told labs that they would offer emergency approval for their kits - but placed such onerous requirements on getting the approval that almost no labs could achieve it (for example, you needed to prove you’d tested it against many different coronavirus samples, but it was so early in the pandemic that most people didn’t have access to that many). Then they approved a CDC kit which that the CDC could send to places other than their headquarters, but this kit contained a defective component and returned “positive” every time. The defective component was easy to replace, but if you used your own copy like a cowboy then the test wouldn’t be FDA-approved anymore and you could lose your license for administering it.

A group called the Association of Public Health Laboratories literally begged the FDA to be allowed to deploy the COVID tests they had sitting on the shelf ready for use. The head of the APHL went to the head of the FDA and begged him, in what they described as “an extraordinary and rare request”, to be allowed to test for the coronavirus. The FDA head just wrote back saying that “false diagnostic test results can lead to significant adverse public health consequences”.

I repeat my comment on tests for diagnosis vs. tests for public health. Let us add a first note on FDA skewing its decisions way outside of actual science in order to try to manage public psychology. 

So everyone sat on their defective FDA-approved coronavirus tests, and their excellent high-quality non-FDA approved coronavirus tests that they were banned from using, and didn’t test anyone for coronavirus...

 More good stories follow

Vaccines? Same story a year later. Lessons learned...?  

Although the FDA did kill thousands of people by unnecessarily delaying COVID tests, at least it also killed thousands of people by unnecessarily delaying COVID vaccines. I’ll let you click on links for the details (1, 2, 3, 4, etc, etc, etc) except to remind you that they still have not officially granted full approval to a single COVID vaccine, and the only reason we can get these at all is through provisional approvals that they wouldn’t have granted without so much political pressure.

My emphasis.

Costs and benefits

I worry that people are going to come away from this with some conclusion like “wow, the FDA seemed really unprepared to handle COVID.” No. It’s not that specific. Every single thing the FDA does is like this. Every single hour of every single day the FDA does things exactly this stupid and destructive, and the only reason you never hear about the others is because they’re about some disease with a name like Schmoe’s Syndrome and a few hundred cases nationwide instead of something big and media-worthy like coronavirus. I am a doctor and sometimes I have to deal with the Schmoe’s Syndromes of the world and every f@$king time there is some story about the FDA doing something exactly this awful and counterproductive. 

Read on for the great story of Short Bowel Syndrome. Many babies died from FDA inaction. 

Now to the main point. Scott reacts to an Atlantic article accusing the FDA of not being cautious enough. Because, yes, it occasionally does approve drugs - Aducanumab for example -- that seem not to be effective, and sometimes dangerous. It's going to happen. The best clinical trials have 1,000 or 10,000 patients. You're not going to find 1/100,000 side effects, even lethal ones. 

Well, now we're in to the land of probabilities, costs and benefits, which our administrative state is particularly bad at handling, especially when political pressure gets brought to bear on one side or another ("approve this experimental drug now, it's our only hope!") and the media circus naturally wants heads on platters for any ex-post mistake, even ones like 1/100,000 side effects that might be worth saving 1000 people. Where is the FDA? In Scott's words, 

How many lives would have been saved if good drugs had been released a few years earlier, versus how many lives would have been lost by missing dangerous side effects? I think the current state of the art is something like Isakov, Lo, and Monterhozedjat , which finds that there are a tiny few disease categories where the FDA might be slightly too aggressive, but that overall the FDA is still much too conservative.

And these kinds of analyses, while good, can only count the drugs we know about. The real cost is the thousands of life-saving medications that are stillborn because nobody wants to go through the literally-one-billion-dollars-per-drug FDA approval process.

The latter is the economists' point that we make far too seldom in many contexts. Don't measure the cost of regulations by the cost to current businesses and consumers. Measure the costs of all the businesses that shut down, that didn't get started, all the consumers not served at all. So, to Scott

..if anyone ever gives you control of the FDA Lever, you should definitely absolutely for the love of God push it as far toward LESS as it is possible for it to go - I think this is what all those cost-benefit analyses the epidemiologists and economists publish are telling you, and it’s also what my common sense and ethics tell me. But I have to admit that this isn’t costless. It’s going to let a lot of crappy drugs like aducanumab get through and give people false hope.

Incentives 

And now to the most important point. Why does the FDA behave this way? They're not bad people and they're not dumb. They are stuck in an institutional setup that gives them horrible incentives. This is really key. You cannot reform the system without fixing the incentives. Our politics and media love a morality play of good and bad people. This isn't it. 

I don’t think individual FDA bureaucrats, or even necessarily the FDA director, consistently make stupid mistakes. I think that given their mandate - approve drugs that definitely work, reject ones that are unsafe/ineffective, expect people to freak out and demand your head if any unsafe/ineffective drug gets through, nobody will care no matter how many lifesaving treatments you delay or stifle outright - they’re doing the best they can... 

That nails the incentives pretty nicely! Had the FDA done as I ask, recognize that we need fast, cheap, not so reliable tests and quickly certified the paper-strip tests for over-the counter sales, someone would have gotten a false result, died, and the FDA would have been hauled in to Congress and raked over the media coals. 

Is it Congress' and the higher up Administration's fault? 

And it’s hard to even blame the people who set the FDA’s mandate. They’re also doing the best they can given what kind of country / what kind of people we are. If some politician ever stopped fighting the Global War On Terror, then eventually some Saudi with a fertilizer bomb would slip through and kill ~5 people. And then everyone would tar and feather the politician who dared relax our vigilance, and we would all restart the Global War On Terror twice as hard, and drone strike twice as many weddings. This is true even if the War on Terror itself has an arbitrary cost in people killed / money spent / freedoms lost. The FDA mandate is set the same way - we’re open to paying limitless costs, as long as it lets us avoid a very specific kind of scandal which the media will turn into 24-7 humiliation of whoever let it happen. If I were a politician operating under these constraints, I’m not sure I could do any better.

So the long-term solution is to become a different kind of country and different sorts of people - eg raise the sanity waterline.

Since "unsafe at any speed," our polity has not handled cost-benefit well at all. Perfectly safe is not the answer. We allow horrendous risks of well-known activities -- we don't shut down car driving, smoking, or alcohol -- but anything new faces immense obstacles. If you have to ask the answer is no. Nuclear power and the awful incentives of the Nuclear Regulatory Commission offer an eerily parallel example.  

But until then, are there any small changes that would help around the edges?

Reforms 

What are small, doable changes in the incentives that one could imagine our political system offering a run-amok regulator like the FDA?

1. Unbundle FDA approval.  

Roughly speaking, we live in a system in which if the FDA approves a drug, then insurance companies and the government have to pay for it. (Scott is good at some of the nuance in this statement.) 

when I talk about unbundling FDA approval, I mean that instead of the FDA approving the following bundle of things…

1. It’s legal for doctors to prescribe a drug

2. It is mandatory for insurances to cover a drug.

…the FDA can say one of those two things, but not the other.

Right now these decisions are so charged because, if something doesn’t have FDA approval, then even someone who desperately wants a medication, and has researched it very hard, and is being treated by the world’s top specialist in their condition, and is willing to pay for it with their own money - can’t get it. But if something does have FDA approval, then any moron can get it, just because they saw a TV ad saying it was the hot new thing, and the government/insurance/other patients/Yagmuk will be forced to cover the entire price. 

This is part of a larger problem, that no politician can dare say that people should have to actually pay for anything with a "health" label. The sadly illogical but ironclad corollary, if you think of it, is that nobody is allowed to pay. Because then health would be "inequitable." Oh, and nobody is allowed to feel the incentive to take a job because it gives you money that might come in useful if you get sick...

Back to the topic, however, this strikes me as a very important reform. Institutions are better if they have limited purposes. The FDA is poorly designed to make the choice "should insurance companies be forced to pay for x drug." That leaves open the hard question just who does make that choice, which is I think why it has fallen to the FDA. The rot has to go somewhere. 

2 Information, a star system, not yes/no

Here I go beyond Scott a bit. The FDA should stick only to providing information about safety and efficacy of drugs, rather than make a yes/no decision about safe and effective vs. unsafe or ineffective, and  it is  illegal  to use any non-approved drug. Moving to the role only of information provider, with nuanced information (as information is) would help a lot on the first idea too. 

Scott:

...Grant drugs one-star, two-star, etc approvals. Maybe one-star would mean it seems grossly safe, the rats we gave it to didn’t die, but we can’t say anything more than that. Two-star means it’s almost certainly safe, but we have no idea about effectiveness. Three-star means some weak and controversial evidence for effectiveness, this is probably where aducanumab is right now. Four-star means that scientific consensus is lined up behind the idea that this is effective, this is probably where the COVID vaccines are right now. Five star is the really extreme one where you’re boasting that Zeus himself could not challenge the effectiveness of this drug - the level of certainty around MMR vaccine not causing autism or something like that.

And, I would add, very good information about safety. Not perfect safety, as no drug has that, but the list of potential side effects, frequency, etc. is pretty well known. 

Scott:

Then you could attach different legal rights and requirements to each of those. Maybe the world’s top specialists could start prescribing a drug once it has two-star approval, regular doctors could prescribe it with three-star, drug companies can’t start advertising it until it’s four-star, and insurance companies are mandated to cover it once it’s five-star.

John: Let's not jump to laws. Just put this information out there and let people, doctors, insurance companies and lawyers figure it out. 

People are really scared of this solution, because it introduces choice into this system. If you say that insurance companies are allowed to cover a certain drug, but not forced to do so, then different insurances will cover different drugs, and you’ll have the usual capitalism / free market thing. Patients will have to choose which insurance to get without necessarily knowing very much about medicine, and maybe companies will try to trick or exploit them, and maybe the patients will make the wrong choice.

John: Oh no. Choice. Just like everything else in life. Free to choose, and free to make mistakes. What Scott gets wrong is that there already is choice in the system. Either the FDA and medical bureaucracy makes these choices, or we do with the best information the FDA can give us. Given how well the paternalistic bureaucracy has been doing about making choices, I would ask just who is the more behavioral and making poor choices? 

I can’t stress enough how often the current system results in things going wrong that nobody thought of because the things are too stupid for anyone to even imagine they were possible.

Like I said. 

3. Stop managing public psychology

We saw with the slow approval, slower still final approval and the J&J pause the FDA doing things that were scientifically indefensible in order to try to massage public opinion of the FDA. Scott: 

... When the FDA delayed COVID vaccine approval, or refused to approve various brands of COVID vaccine, or suspended the distribution of COVID vaccines for bad reasons, it always had the same excuse - what if there was a side effect? The problem isn’t that people might die - people were definitely dying from their decision to delay/ban vaccines. 

Though it's not clear the FDA ever puts "people might die waiting for us to approve a drug" very high in its priority list, but perhaps they secretly do

The problem was that people might stop trusting the FDA. They would say “the FDA allowed me to take this drug, but it was dangerous, screw them, I will never take an FDA-approved drug again in my life and only use homeopathy from now on.” The FDA and medical policymakers live in terror of this scenario. They feel like if they ever allow even one bad drug through, then in the eyes of the public all kinds of anti-vax hysteria and vaccines-cause-autism bullshit will be retroactively justified, and public health officials will never have any authority ever again. If you model all FDA/CDC/etc policy as an attempt to avert this outcome, your predictions will be right more often than not.

This is another thing I’m pretty sympathetic about - social trust is a valuable resource. But it also means that public policy will forever be held hostage to the whims of the stupidest person around. Every time someone sneezes, the FDA will ban whichever brand of COVID vaccine they got - because if they didn’t, then stupid people might believe the FDA didn’t take vaccine side effects seriously, and then those stupid people would stop getting vaccines and die. This policy has led to our current situation, where either everyone has to be miserable because of stupid people’s choices (eg everyone has to wear masks forever because a few people won’t get vaccinated) or we get a strong anti-freedom lobby because allowing anyone any freedom means that the rest of us have to suffer for their stupid choices. 

I have a different take. What is the scientific evidence that the FDA's psychology management works at all? J&J was a fiasco. By "pausing" they did not give a widespread feeling of just how good the FDA is and therefore raise trust in the drug. They had exactly the opposite effect. People inferred that J&J is unsafe, and they lowered trust in this very nice vaccine to this day. (My latest reading says J&J is just as effective, but we only take one dose because that's how the trial was set up and you can never change dosing in light of experience without more trials, another big problem that Alex Tabarrok has taken on. J&J also is easy to store and transport.) 

I'd rather that efforts to affect public psychology must pass the same three-phase randomized clinical trial as do medicines!

But seriously, going to simply an 5 star level information provider rather than enforcer of laws against using drugs even if you pay for them, and czar of what insurance and government must pay for, would allow great progress here too. 

4. Budget FDA by drugs approved weighted by time and health/life savings 

The FDA does not see any financial or institutional benefit to speeding along the process of lifesaving drugs. Its purpose, remember, is to forbid us from taking drugs. Well, let's give them an institutional incentive, balancing out all the ones so far to say no. 

In regard to nuclear power I quoted Jason Crawford, 

The individuals who work at NRC are not anti-nuclear. They are strongly pro-nuclear—that‘s why they went to work for a nuclear agency in the first place. But they are captive to institutional logic and to their incentive structure.

The NRC does not have a mandate to increase nuclear power, nor any goals based on its growth. They get no credit for approving new plants. But they do own any problems. For the regulator, there‘s no upside, only downside. No wonder they delay

Further, the NRC does not benefit when power plants come online. Their budget does not increase proportional to gigawatts generated. Instead, the nuclear companies themselves pay the NRC for the time they spend reviewing applications, at something close to $300 an hour. This creates a perverse incentive: the more overhead, the more delays, the more revenue for the agency.

Ditto FDA

We don't need "more" or "less" regulation. We need smarter regulation. And that will come by improving the incentives faced by regulatory agencies.