_Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials_
https://pmc.ncbi.nlm.nih.gov/articles/PMC300808/
"Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute."
We could be much more flexible in our approach to things if we would un-ban things. For example after phase 1 trial or based on sufficient observational evidence, things can no longer be banned but have a higher standard for "insurance is now required to cover it".
This is not happening.
> I was surprised to find that they usually discard papers based on observational evidence wholesale.
He he...welcome to the real world!
I noticed that as well. My eyes feverishly scanning the previous paragraph for the definition.
Don't forget that RCTs are very far from perfect and issues -- sometimes literally fatal issues -- have later turned up via observational evidence in large cohorts. Vioxx, for instance. Many others.
I believe, without the tiniest shred of doubt, that the only trials drugs need to go through are initial safety/toxicity trials (phases 0/1) and that everything else would be much better left to access+observation.
Should we demand an RCT before we accept evidence? Of course not. At some point you do have to make a choice on things.
And it should be noted that most drugs do have early cutoff criteria if the evidence is strong enough that it is working. It isn't like people are wanting to withhold good treatments from the world. Adding controls and randomizing them, though, has proven to be highly effective at helping progress.
If you have enough data, you can smooth out individual fluctuations due to things like drug interactions, non-compliance, etc. (And indeed you might discover drug interactions!) Observational trials ultimately mirror how drugs are used in the real world.
> "Adding controls and randomizing them, though, has proven to be highly effective at helping progress."
I would argue just the opposite. Demands for increasingly byzantine trials have ballooned the costs associated with drug development, and have slowed things to a crawl. There's a reason the field's golden age was in the 1940s and 1950s, and it's not just "low hanging fruit." Today nobody in their right mind wants to work in drug development when they could work in tech or even finance.
Again, it is off to think that one is automatically superior to the other. Certainly to the exclusion of the other. And that is what feels off with the framing of the parent post. I am perfectly fine saying you should use both observational and controlled trials. But I think it is also wrong to think you don't have to build experiments to test interventions.
This is why you put metrics in your service code. So that you can observe them behave and look for things to change. This is also why you do test cases on your code, so that you can specifically target your change.
Now, I fully back the idea that just A/B testing something doesn't automatically mean you learn something true. But neither does observing a strong outcome on uncontrolled data.
"Large controlled experiments are costly and can hurt people who opt-in to informed consent. Instead, we should do significantly, significantly larger experiments, with undefined success/failure conditions, and no informed consent."
Insane opinion
Even if that is true, is it an intrinsic problem with trials or just bad regulation? If it is the latter then you need to change the regulations? Is the problem global - is every regulator everywhere demanding byzantine trails?
Yes, that was because of things like
https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment
https://en.wikipedia.org/wiki/Stateville_Penitentiary_Malari...
https://en.wikipedia.org/wiki/Operation_Sea-Spray
https://en.wikipedia.org/wiki/Nazi_human_experimentation
I understand that certain people are salivating at the thought of a return to those times; I'm not one of them.
Wow.
This is silly.
FDA has essentially one requirement: prove that your drug is safe and effective.
The reason trial designs get more and more byzantine is because the drugs themselves work less-and-less well. They're far more nuanced and precise. The experiments have to be extremely well-controlled, and then this has to balance against cost/timeline of the trial, and that's why sponsors choose to use byzantine trial designs.
This is especially great because it puts anyone who actually wants to actually make a working drug at a significant disadvantage. It'll take them longer to get to market, cost them a billion dollars more, then their medicine gets to sit next to a thousand variations of "Vitamin C for Leukemia" that all cost a lot less.
There would be virtually no incentive for anyone to make an actual drug.
> Postmarketing surveillance can easily determine what's effective and what's not, and medical orgs can adjust.
"Easily" is doing a ton of work. Postmarketing surveillance can sometimes give low-confidence signal as to what's effective and what's not.
This is just nonsense. First, everyone in a trial is informed of the situation. It's not "unethical" unless you lie about it. If you participate in a trial, you do so knowing that you might not get the experimental drug. It's a selfless, honorable thing to do, and we shouldn't be framing it as some kind of scam.
Second, we don't give terminally ill people "fake treatment" (placebo trials). We give them current standard of care. Giving someone a placebo trial doesn't prove anything that would change clinical practice, because you want to know if the drug works better than what is out there today. Rarely is that standard of care "nothing", and this (bad controls) is actually a primary reason that a lot of drug company trials are rejected by the FDA.
If I didn't see the Wall Street Journal editorial board repeating the same garbage in defense of patent medicines, I'd write you off as simply having a sophomoric understanding of how trials work. I'm convinced that someone is driving this absurd narrative.
I wish people would stop saying this. First, controls aren't necessarily "fake treatment", they are often compared to other standard treatments.
Second, the treatment being tested can actually harm the patient more, therefore the people receiving your alleged "fake treatment" can actually come out better off. Which is the "fake treatment" now?
I don't disagree with your final point, but mainly with this increasingly pervasive and wrong framing of RCTs.
This can be more subtly critical than it might seem, in that even if you can manipulate some proxy, often that proxy is insufficient in actually representing the phenomenon of interest, or the conditions under which they actually occur.
I often use the example of videogames and aggression. There were plenty of experimental studies of this but it was always questionable whether lab-induced anger is the same thing as, say, the sort of violence we generally are concerned about societally.
I generally have tried to teach students that experimental designs when done right provide powerful causal evidence of something, but often with limited generalizability; observational designs in contrast provide powerful generalizable evidence of some kind of association, but often with limited certainty about the causal pathways involved.
I've been in a department that was rabidly experimental in its focus and it always seemed sort of short-sighted, because people were idolizing RCTs with proxy manipulations that had questionable generalizability to the real-world phenomena they were trying to model.
Ideally you'd bring both experimental and observational evidence to bear on a question. Your conclusions should be robust to different types of designs.
OP suggests that alternative methods like target trial emulation, propensity scoring and double machine learning can be used to approximate the conditions of an RCT using existing data. In saying so he gives away the tell, which is that RCTs are the standard being aspired to.
Observational data may well be undervalued. But RCTs are still the gold standard.