The majority of funding for studying how organisms work comes under the auspices of attempting to “cure” or ameliorate the symptoms of a perceived “disease”. Specifically with regard to neuroscience function, the psychiatric “ordered/disordered” conceit drives an overwhelming amount of the work in the field.
We’ve invested hundreds of billions chasing down “cures” for psychiatric descriptions, and often show fantastic results in lab settings. Every few years we cycle to another miracle cure that paper after paper shows amazing results in lab settings, however when exposed to the real world turns out to be barely better (and sometimes worse) than placebo.
For any given treatment there’s likely a pretty solid level of research which demonstrates the value of it, whether it be transcendental meditation or ketamine/hallucinogens. In spite of all this, the incidence and prevalence of the very things these treatments seek to address continue to advance.
Underlying this disconnect is the difference between “efficacy” and “effectiveness”, or whether something works in study settings vs. whether something works in the real world.
In study settings, the variables are tightly controlled, the candidates are hand picked, and length of time is relatively short. Efficacy only needs to demonstrate the thing under study can work.
In the real world, we have no such control over variables or candidates. We also need to be responsible for much longer periods of time, potentially the subjects entire lifespan. Effectiveness needs to demonstrate that it does work.
There are quite a few more traditional treatments which actually show effectiveness, that is they’ve been used to treat particular symptoms or conditions longitudinally for a long period of time, however fail to show efficacy in tightly controlled lab conditions because the study setup variables were not able to demonstrate it.
In the same vein there are many studies which show outstanding efficacy (e.g. SAINT for “depression” or MDMA for just about anything), but completely fall apart when exposed to the real world.
It’s important to note that whenever we review work which makes outstanding claims about the efficacy of a process in a study, or shows a vanishingly small p-value (meaning the null hypothesis should be almost impossible to disprove), that efficacy and effectiveness are two different things.
This conceit carries over to studies which attempt to demonstrate biological function especially “knockout” and optogenetic work. That the carefully controlled lab settings are able to produce a consistent effect does not demonstrate how effective the understanding is, or how descriptive it is of real world mechanics.
Generally, until we are able to get past this binary I turned it on and it did this, I turned it off and it didn’t do this type of study construction, we are going to continue to tread water in creating effective understandings of nervous system function.
We are still in a state where every single piece of work should be regarded skeptically, no matter how many dopamines per neuron upregulations per vesicles happen to align with our current beliefs. Efficacy is not evidence in and of itself until it demonstrates effectiveness.
Efficacy versus Effectiveness – Describing medicine study results as whether they do “more harm than good” is a pretty jarring perspective.
A Primer on Effectiveness and Efficacy Trials – RCTs are better than nothing, but still lack the ability to describe effectiveness.