Monday, January 24, 2011

The Scientific Method & Its Limits - The Decline Effect

First up, for all the climate scientists out their still publishing their experiments while withholding the data and computations necessary to reproduce their results, here is a description of the scientific method and why it is used:

. . . Different scientists in different labs need to repeat the protocols and publish their results. The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.

. . . [R]eplication is what separates the rigor of science from the squishiness of pseudoscience . . .

Actually, I think the word "pseudoscience" is an understatement. If we are asked to believe that which cannot be replicated, then we are being asked to accept results based on faith. That crosses the clear line demarking science from religion. And indeed, it is this litmus test that puts much of climate science in the latter category.

At any rate, the above comes from a fascinating article in the New Yorker by scientist Johah Lehrer, in which he discusses how "scientific truths" established under ridgid testing, are losing their validity over time:

But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.

For many scientists, the effect is especially troubling because of what it exposes about the scientific process. If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved? Which results should we believe? Francis Bacon, the early-modern philosopher and pioneer of the scientific method, once declared that experiments were essential, because they allowed us to “put nature to the question.” But it appears that nature often gives us different answers.

This is a question with significant ramifications for science and how much trust should reasonably be placed in any particular finding as establishing an immutable truth. Lehrer goes on to suggest that both individual bias and "publication bias," are the major factors in skewed science. He goes on to discuss one recommendation:

According to Ioannidis, the main problem is that too many researchers engage in what he calls “significance chasing,” or finding ways to interpret the data so that it passes the statistical test of significance—the ninety-five-per-cent boundary invented by Ronald Fisher. “The scientists are so eager to pass this magical test that they start playing around with the numbers, trying to find anything that seems worthy,” Ioannidis says. In recent years, Ioannidis has become increasingly blunt about the pervasiveness of the problem. One of his most cited papers has a deliberately provocative title: “Why Most Published Research Findings Are False.”

The problem of selective reporting is rooted in a fundamental cognitive flaw, which is that we like proving ourselves right and hate being wrong. “It feels good to validate a hypothesis,” Ioannidis said. “It feels even better when you’ve got a financial interest in the idea or your career depends upon it. And that’s why, even after a claim has been systematically disproven”—he cites, for instance, the early work on hormone replacement therapy, or claims involving various vitamins—“you still see some stubborn researchers citing the first few studies that show a strong effect. They really want to believe that it’s true.”

That’s why Schooler argues that scientists need to become more rigorous about data collection before they publish. “We’re wasting too much time chasing after bad studies and underpowered experiments,” he says. The current “obsession” with replicability distracts from the real problem, which is faulty design. He notes that nobody even tries to replicate most science papers—there are simply too many. (According to Nature, a third of all studies never even get cited, let alone repeated.) “I’ve learned the hard way to be exceedingly careful,” Schooler says. “Every researcher should have to spell out, in advance, how many subjects they’re going to use, and what exactly they’re testing, and what constitutes a sufficient level of proof. We have the tools to be much more transparent about our experiments.”

In a forthcoming paper, Schooler recommends the establishment of an open-source database, in which researchers are required to outline their planned investigations and document all their results. “I think this would provide a huge increase in access to scientific work and give us a much better way to judge the quality of an experiment,” Schooler says. “It would help us finally deal with all these issues that the decline effect is exposing.”

I couldn't agree more. And the first place to start is in the pseudoscientific area of climate science.

11 comments:

suek said...

Sounds a bit like the reliability of the results we get from the CBO...

GIGO

OBloodyHell said...

> Actually, I think the word "pseudoscience" is an understatement

Indeed, beware of climate ducks: Quack Quack.

OBloodyHell said...

"The Truth Wears Off"?

What a friggin' CROCK.

1) This demonstrates, in fact, that the Method -- and its lynchpin, "reproducibility" ARE IMPORTANT AND VALID to learning to distinguish Truth from Wishful Thinking and Flights Of Fancy.

2) "When a large number of studies have been done on a single subject, the data should follow a pattern: studies with a large sample size should all cluster around a common value—the true result—whereas those with a smaller sample size should exhibit a random scattering, since they’re subject to greater sampling error"

In short, the validity of extending and depending on data from small data sets "proven" using statistical techniques, like, oh, the F-Test, which I recall being told in Intro to Stats could produce reliable conclusions from as little as 7-8 data points is notably specious?

Gosh, 7-8 data points doesn't necessarily reflect a useful statistical universe!?!?!

Wow, whoodathinkit!?!?!

Owww... excuse me, it HURTS when my eyes roll back in my head that far.

3) All the examples listed are "soft" sciences, or, at the least "fuzzy" sciences, like climate science and ecology, in which the so-called "scientists" are already known to play havoc with terms, like using "species" when what is actually meant is "variety".

OBloodyHell said...

> Palmer has since documented a similar problem in several other contested subject areas. “Once I realized that selective reporting is everywhere in science, I got quite depressed,”

Nice "extension" there -- from "a few contested subject areas" to "everywhere in science".

No, just the ones which allow for fuzzy result interpretations, Dr. Palmer. Engineering, Physics, Math, and even Chemistry ... they don't see crap like this all that often.

> Between 1966 and 1995, there were forty-seven studies of acupuncture in China, Taiwan, and Japan, and every single trial concluded that acupuncture was an effective treatment. During the same period, there were ninety-four clinical trials of acupuncture in the United States, Sweden, and the U.K., and only fifty-six per cent of these studies found any therapeutic benefits. As Palmer notes, this wide discrepancy suggests that scientists find ways to confirm their preferred hypothesis, disregarding what they don’t want to see. Our beliefs are a form of blindness.

Yes, there's no difference in the quality of the science being done. All the groups are the same in their devotion to The Truth and their care in avoiding personal biases entering the results.

Yeesh. Another ridiculous equivalence.

OBloodyHell said...

> Because most of these studies were randomized controlled trials—the “gold standard” of medical evidence

Uh, no, the DOUBLE BLIND test, where the evaluators generally don't know what they are attempting to find out or anything much about the experiment's null/non-null hypothesis is the "gold standard" of ALL sciences.

OBloodyHell said...

> in which he discusses how "scientific truths" established under rigid testing, are losing their validity over time

The thing that concerns me is that this sort of thing, as quoted, undermines public belief in the Scientific Method as a whole, rather than teaching people to mistrust the results in **certain easily manipulated fields of study**... such as climate science.

It supports belief in the notion that Science is less valid and less reliable than it is, and/or that the notion of "Truth" is itself invalid or unreliable or subject to change.

Whether the author meant for it to be taken that way, the title:
The Truth Wears Off
Is there something wrong with the scientific method?

assists in the postmodernist attack on our Greek cultural heritage.


Not everyone will read it, but a lot of them will read that title, the blurb, and the first couple paras, you can bet your ass.

This quote is damned sure going to be remembered:
But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable.

That these claims in question are claims made by fools and self-serving charlatans, most importantly in fuzzy fields of study lacking effectively rigorous rules for evaluating data isn't going to be a part of what the readers "learn" -- that's buried in four-syllable words on page four.

Not surprised a libtard postmodernist rag like The New Yorker published this.

GW said...

OBH - lol, I think this one rattled your cage. I actually chose to blog this one for the, I thought, reasonably good description of the scientific method. I included the rest because it gets around to talking about the need for much tighter experimental controls, opening up all data, etc. on the net, and the need for fairly large data points. Obviously, I don't have your depth of statistacal knowledge, though the above points seem to be obvious even to a layman. All are implicated in the not fuzzy, but rather opaque, world of climate science.

O Bloody Hell said...

The Scientific Method & Its Limits - The Decline Effect

The issue is that the article itself is largely mistitled -- it's not about doubts regarding The Scientific Method itself
at all, but more about the kind of abysmally lousy "quality control"
over its application in certain "softer" fields of study.
;-)

O Bloody Hell said...

GW, you should realize -- you and I likely have similar takes, I'm just rather annoyed on the whole with the broadband attacks on Greek Ideas and Thought which are at the core of postmodern liberalism.

The left is literally a vile, loathsome cancer eating away at the very basic concepts that made Europe, especially England, and most especially America, the success and the shining light they are in humanity's history.

If I could push a button, commit genocide, and get rid of the lot of them, painlessly, I'd gladly take my place in Hell for having done so.

OBloodyHell said...

It's been a while, but I would call attention (H/T: Coyoteblog) to

The Reproducibility Project

There's hope to reclaim Science from the postmodernists quacks and charlatans, after all...

OBloodyHell said...

Another while, but this appears to be of interest and relevance, since it's "peer reviewed" (heh):

PLOS|Medecine: Why Most Published Research Findings Are False