Papers report very early stages of knowledge formation. They typically contain an argument: what was done, why, what was expected to happen, what actually happened, how did the authors interpret that, and why. Rather than describing the world directly, a paper describes a belief about the world, and justifications for the belief.
Published results are often contradictory, because individual papers are unreliable. Something may have gone subtly wrong, the interpretation could lack nuances, some key understanding may still be missing, or the authors may have just been unlucky. When an expert reads many papers on related topics, the arguments shape their beliefs. Eventually a scientific consensus may emerge, which is the next (but still an insufficient) step towards reliable knowledge.
This is something I bookmarked (in part because I'm an admirer, but also because it made sense and squared with my limited experience of academic peer review):
I think people get in trouble when they think academic publication has value in itself. Ideas become valuable when people care about them. Publication is one particular path to reach a community of readers but it doesn’t make your ideas matter.
Isn't this a kinda narrow view of what's valuable? Sure, it's valuable for your reputation or your academic career only if it becomes widespread.
But if something has predictive power, it's valuable no matter how many other people know it. In some cases, it's way more valuable when it is still unknown, because nobody has had the chance to capitalize on it yet.
I'm not familiar with the person behind the tweet, so I don't know any more than what the quote says. But I'm saying that if an individual paper says a thing, and that thing ends up being useful, then how is that not valuable? It could end up being wrong, but so could anything else you pick up from anywhere.
In some instances, it won't end up being useful because it's only relevant to someone in a lab full of million dollar equipment. But that'll only be the case sometimes.
Some papers are super important! Some papers aren't. (from the ice age... to the dole age...) "Being a paper" isn't a very meaningful metric. That's all that's being said.
I think this is a problem if you take one paper and then you weight its conclusion at 100%. It's less so if you take all of the conclusions and realize they all have a weight (maybe 0). But I think that's true for every source of information.
My approach is to avoid framing things in terms of "now I know x is true", and instead look at it in terms of "someone believes (or wants me to believe) x is true". I then weigh all of those beliefs as I observe the world around me. I resign myself to never really knowing anything myself, but having some idea of what different schools of thought are on a given topic.
If you move your information gathering further along the chain, to maybe a text book, or some expert's twitter feed, why should these things be more reliable? If they've correctly come upon the consensus view, you still have to consider that informing you of the truth night not be their first priority. They could be after money, advancement in their field, political agendas, and (though less likely) they still could be just plain wrong.
So they're more likely to know the truth, but you're still unable to evaluate whether they're giving it to you or not.
Published results are often contradictory, because individual papers are unreliable. Something may have gone subtly wrong, the interpretation could lack nuances, some key understanding may still be missing, or the authors may have just been unlucky. When an expert reads many papers on related topics, the arguments shape their beliefs. Eventually a scientific consensus may emerge, which is the next (but still an insufficient) step towards reliable knowledge.