When I started reading the article I thought the title was a bit click-baity for Science. But then I noticed that the theory it refers to was not the amyloid hypothesis itself, but the toxic oligomer hypothesis that emerged later. That theory is pretty much the main potential explanation on why every single drug targeting Aβ failed that still keeps Aβ fibrils relevant. It's a very convenient theory because it keeps the main original observations about the fibrils relevant while explaining why therapies that target them don't work.
One part that is really important that is mentioned in the middle of the article is that these systems are very difficult to handle, and it's almost impossible to make many of them nicely reproducible. Fibril and oligomer formation depends a lot on the environment and reacts to tiny differences.
I find this kind of fraud deeply frustrating, there is so much wasted effort in the wake of faked high-profile results.
It's more insidious than that. In a field where the core subject is difficult to handle it is nothing unusual if a different lab cannot recreate known experiments on their first try. This doesn't mean anything is wrong with the original research, it might just mean that the second lab didn't control all the variables.
The part that is difficult and annoying is that the variables are not necessarily known, it takes a lot of time and experiments to actually nail down those. And even then for some sensitive stuff every detail can matter, like the exact type of tube you did the experiment in or the vendor, batch and age of every chemical you used. Very often you can control this enough to have consistent results for a set of experiments, but that kind of stuff is really hard to control across different labs.
The social default is in some fields where work is hard to reproduce is they still get into good journals and are believed up front. Usually this is fine. A lot of work that is hard to reproduce, but are relatively important or open up many research directions, do eventually get reproduced, though it might take a year or more. And usually the first party does their best to try to help reproduce the results.
I wonder what it would look like if you’re not allowed to cite papers in the field unless there’s been 5 independent labs that have managed to reproduce the result and all the labs that reproduce get to share in the credit (eg if there’s a Nobel prize). Would that change the incentive structure and create better outcomes? Science isn’t science until you have a repeatable process in place where observations match outcomes and the underlying theory. Someone proposing a bold new theory without the ability to test it usually gets significant credit even if posthumously for being a bold visionary which seems to have been sufficient throughout time as a motivator for the really foundational scientific victories.
The effort required to reproduce a result, especially an exotic one, is often prohibitive. The reward for successful reproduction is near zero, and if you fail, it’s often absolutely zero - you are not going to be able to contest a result by a failure. The original authors can simply say that your reproduction was flawed, if they need to answer at all.
A journal that requires five confirmations to cite a work is a journal that will fast have zero submissions.
The main reason people attempt reproduction is to continue that line of research, and that often starts with reproduction plus a slight tweak. This is why you find a lot of papers like this. This is also how irreproducible results become sort of known about but not challenged in many fields.
So what does the incentive structure look like? Do you get tenure for running a lab that replicates? Are your postdocs in line for tenure track positions? How is it funded: do you put the secondary replications into the original grant? What happens if you can't find X number of labs to run the test?
It's an interesting idea, but it all comes down to how the incentive structures happen.
Presuming the original lab didn't also intentionally leave out important details to make it harder for others to replicate their research quickly. Why list all the details needed to replicate your research and possibly let other labs catch up and take your next grant? Much better to add a veneer of listing details but leave out a few of the really important ones.
I heard from a chemistry researcher that it is also common in that field to give vague descriptions of synthesis procedures in order to allow authors to churn out more papers.
From reading online it feels that most important thing is to be first. And prevent anyone else even from potentially moving forward on your topic. The actual scientific progress is secondary.
yeah, I was just asking why you thought it was appropriate to extrapolate (if you had any reason to believe that the shady incentives you mention, translate to a different field, versus them being completely different in this regard)
To me this is why the emphasis on “reproducibility” is misguided. You might end up failing because you lack the technical capability. Conversely, you might “succeed” by making a mistake. We should be focusing on coming up with new ways to test the same hypothesis.
The null hypothesis tells us that if we can reproduce repeatedly and reliably then we can actually use even relatively small number of reproductions to raise the net significance of the result (which also diminishes the ability for fraud to work as it requires a larger conspiracy instead of people just accepting people at their word).
If you think reproductions have failed then it’s on you to try to have a cogent explanation for why if you’re personally motivated by a belief that the experiment is actually valid.
The particle physics folks also have a different approach these days where they blind the ability to see results and do multiple cross checks of each other’s results to validate the data (because the underlying experiment is so expensive to run). That can also work in theory although hard to say what the success rate of that approach is just yet.
indeed, the initial experiment is not immune to such concerns, and might itself have succeeded by making a mistake or failed because it lacked the technical capability
Would it be more convincing if labs corroborated other lab's findings? Like Lab A found that XYZ impacts LMQ and Lab B, following Lab A's findings, was able to replicate the finding within some margin.
I worked for a pharmaceutical company that tried to open a second factory producing more of the same thing that it was already successfully making and it never worked. They built the entire factory, it didn't work, they spent years and millions of dollars trying to debug it, it simply never worked out. They ended up just essentially running the US factory at three shifts to meet the increased demand. To this day I don't think anyone knows why the European production line never worked. Things can fail to reproduce without fraud.
Very fascinating. Were you at least able to pinpoint a part of the production process that produced different results, or was the difference only visible once the final product was "assembled"?
I was not closely involved with the project. They essentially built the first one to "spec" like "use this kind of steel in the tanks" and the remediation process was essentially tearing those out one by one and "use the steel that we used in the first tank."
Sounds like a process similar to how you would debug a difficult bug in a big and/or complicated codebase – through trying to isolate the code causing the bug, comparing the buggy code with code that's known to work, replacing other parts of the code with dummy code/mocks etc. And sometimes a bug seems impossible to solve regardless of how much developer time you throw at it, and you just have to ship the product with the bug (seems to be especially common in the game dev business).
It's not even the toxic oligomer hypothesis that is undermined - it is only the role of amyloid beta as a primary driver of Alzheimer's pathology. This isn't even out of nowhere, there's been plenty of data finding no strong association between AB levels and pathology. Most data seems to point to AB being more of a supporting element, with tau amyloids actually being associated with toxicity and resultant functional deficits.
1. Yes. Science has a snowball effect as a result of committee-driven grant decisions. Research in a hot current topic attracts more grant funding more reliably.
2. It does because some fields of science are intrinsically long-duration, highly sensitive to variables, or only testable at scales that exceed our experimental capability. That's just a consequence of physics and natural laws of reality. E.g. nutrition, chemical toxicity, economics, high energy physics.
3. We can cheat and find proxies or more tenable micro-systems to experiment on. But often those have their own problems (e.g. rodent models) or aren't feasible.
Reproducibility is a goal of science. It's not always an achievable goal.
When it's not, we do the best we can, as with drug testing pipelines.
In general, anything that's at the bleeding edge of theoretical understanding and intersects with commercial interest or draws social/media hype has a relatively low probability of standing the test of time. Whether that's replicability, applicability, or just validity.
The most exciting hypotheses tend to go one of two ways. They are rapidly supported with independent evidence, by being applied or replicated. Or they draw a lot of attention and money which helps them persist in spite of, or in the absence of, evidence.
Charlatans, purveyors of snake oil, and (most often) people who for whatever reason don't want to be seen to be wrong - they exist in every walk of life. Science is the same. The incentives in the system strongly select for these people in the 'leading hypotheses' space.
Everything is extremely hype-driven -- it turns out that "cannot confirm X" isn't very compelling for journals, etc. etc. or even the news cycle. Journals, etc. thrive on exciting new findings... and that tends to lessen the critical looks.
Of course, there are lots of other things to this problem: Very narrow fields where people absolutely know who their "anonymous" paper reviewers will be, and so must include even extremely tangential references to those reviewers' papers, etc. etc.
Usually the sciences self-correct eventually[0], but that's only because there is such a thing as objectively verifiable facts and overwhelming statistics in science.
[0] Unfortunately often as slowly as "one funeral at a time" (Max Planck, I think).
1. Twitter bot / misinformation research. A surprisingly large field in which papers are almost never replicable because they don't supply the actual tweets, but only opaque IDs and classifications. Trying to cross-check them is futile because by the time you tried months later many of the accounts have been suspended. They could easily supply the contents of the tweets they scraped along with account metadata, but don't. The few times I did deeper checks of these papers I always found some accounts that were identified as bots but weren't suspended, and on manual inspection were very obviously human. This field also has the problem of being increasingly based on ML pseudo-science.
2. Germ theory! It can't explain several aspects of the epidemiology of respiratory diseases. It's probably not wrong but is certainly incomplete. As we saw with COVID, models based on simple germ theory always make wrong predictions, but this problem pre-dates COVID. It was known for a long time that standard germ theory fails to explain the behavior of influenza, for example, why it's seasonal, why waves peak and enter decline before everyone is infected, why variants disappear totally instead of coexisting, why flu season seems to start everywhere in season almost simultaneously, why there have been outbreaks of respiratory viruses in totally isolated environments like arctic bases, and so on.
The issues here are deeper than reproducibility though. Even if bot papers were reproducible they would still be wrong because their methodologies are invalid and cannot support the stated conclusions. It's important not to get too focused on mere replication.
> 2. Germ theory! It can't explain [...] the behavior of influenza, for example, why it's seasonal,
Huh? I thought that's so well-known (must be, if even I have heard of it) that it goes without saying: Spring-summer, people go out and breathe fresh air a bit apart; autumn-winter, everyone goes inside and coughs their germs at each other.
> why waves peak and enter decline before everyone is infected,
The more people are already infected, the fewer and further between the as-yet-uninfected are, so that seems quite reasonable. (Pretty much inevitable, really, isn't it?)
> why variants disappear totally instead of coexisting, why flu season seems to start everywhere in season almost simultaneously,
See first point above.
> why there have been outbreaks of respiratory viruses in totally isolated environments like arctic bases, and so on.
Someone always flies in supplies...
I mean, WTF -- "Germ theory"? Isn't that what it was called in the 19th century, when there still was any doubt about it? What next; "Gravity is just a theory!"?
That's a popular ad-hoc "street" theory (it's easy to come up with those) but formally speaking that idea has never been tested or proven; it's not a part of germ theory. If you look at epidemiological models, they rarely model seasonality for that reason. The models used for COVID for example, didn't have any notion of seasonality in them for a long time and many still don't, because what you just said isn't actually scientifically accepted.
And it would probably run into some practical problems if you tried to nail it down, like the fact that for many people they are office or factory workers and the amount of time they spend outside doesn't change all that radically during summer, certainly not enough to make the difference between explosive spread and total eradication.
"The more people are already infected, the fewer and further between the as-yet-uninfected are, so that seems quite reasonable"
That's why the wave slows down yes, but again, this ad-hoc notion doesn't work. Try it out on paper or code up a quick epi model yourself and then compare against real world case data. The waves always end long before predicted, even when there are still tons of uninfected people available to infect and people wandering around who should easily infect them.
This is one of the reasons why COVID modelling failed so badly. They coded up the exact simple germ theory model you're describing here, and of course it yields a single giant wave whereas what is seen in reality is a long series of small waves that start and end in ways that aren't predicted by the theory.
"See first point above."
See my reply. I can assure you, that the epidemiology of influenza really isn't as simple as "people going outside in summer". Take a look at some of the research papers from the 80s exploring this topic. There are still plenty of people interacting with each other closely indoors even in summer months, yet flu completely disappears. It's not something that ties in any obvious way back to accepted theory.
"Someone always flies in supplies..."
That's not what "totally isolated" means. In the cases in question there was no contact with the outside world whatsoever. In one case, at a British polar base, they mounted a very thorough investigation after an outbreak of cold virus after 17 weeks of total isolation. They checked if any new supply crates had been opened, etc, but no. They couldn't identify anywhere that new viruses might have been introduced to the base.
One of the more popular sub-theories (though largely ignored by epidemiologists) to try and explain these things is the possibility that many people are continuously infected at a sub-symptomatic level, that the immune system never 100% wipes out the viral infection in these people, just keeps it in check. Then something happens to slightly knock the immune system out of balance for a moment, like a sudden change in temperature, and the infection is able to re-gain a foothold and starts replicating out of control again. So that'd be why people just show up everywhere at once spontaneously infected without any obvious index cases.
The standard of truth in science is reproducibility. But publication usually happens before results have been independently reproduced. Papers are peer reviewed, but generally peer reviewers are just ensuring that the results are sufficiently interesting to warrant publication and there aren't any obvious methodology mistakes.
For the purpose of funding, publication in important journals or conference proceedings is the important thing. And the public generally tends to treat publication as the standard of truth. But publication kind of works on the honor system and is susceptible to fabricated results.
Maybe there should be more funding towards reproducing (or not) important results?
One of the issues is that by chance the models used have an unexpected behavior that seems to confirm the initial hypothesis is sound.
And this is not rare. Sometimes it is fraudulent, people know things are faulty but continue anyway. In other cases it could be by ignorance of the existence of the phenomenon behind the faulty behavior or simply because that phenomenon has not been discovered yet.
One part that is really important that is mentioned in the middle of the article is that these systems are very difficult to handle, and it's almost impossible to make many of them nicely reproducible. Fibril and oligomer formation depends a lot on the environment and reacts to tiny differences.
I find this kind of fraud deeply frustrating, there is so much wasted effort in the wake of faked high-profile results.