The decline effect and the scientific method : The New Yorker

January 30, 2011 at 1:08 pm 4 comments

Education has never been much for replication studies, but given what this article says about psychology, I’d bet that we would have trouble replicating some of our earlier education findings.  I don’t see that this article condemning the scientific method as much as condemning our ability to find, define, and control all independent variables.  The world changes, people change.  Anything which relies on a steady-state world or human being is going to be hard to replicate over time.

Before the effectiveness of a drug can be confirmed, it must be tested and tested again. Different scientists in different labs need to repeat the protocols and publish their results. The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.

But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.

via The decline effect and the scientific method : The New Yorker.

Entry filed under: Uncategorized. Tags: , .

Taking a test helps with learning Closing down computer science at the Minnesota State University

4 Comments Add your own

  • 1. Alan Kay  |  January 30, 2011 at 2:30 pm

    This is the flip side of what Marvin Minsky has called “Physic Envy”. Ultimately physics (and the other physical sciences) work as well as they do because ways are found to go beyond correlations to actual models (with parts that have actual models). At the bottom usually still lies mere correlations, but in between a lot can be accomplished.

    Attempts to be scientific where this is not the case can still work, but at the cost of being much more careful in many more ways than most humans (both investigators and especially funders) can bear.

    For example, our experience with really trying to evaluate curricula in classrooms eventually required at least 3 years to get a bit of substance, and convinced us that most of what is reported in the educational psych literature has “less than a bit of substance”.

    What results is the common bugaboo in soft areas: namely, stories start to replace science, and this makes far too many people far too happy.

    What to do? Engineering before science had a very good answer. It was as critical then as now to not have buildings and bridges collapse. So quite a bit of “testing” plus time plus considerable overengineering allowed (e.g. the Romans) to make buildings and bridges that are completely safe and usable today (e.g. Pantheon and the Merida bridge).

    One could imagine a rule of thumb that the amount of time and effort in investigations is at least inversely proportional to the level of model building the subject permits.



  • 2. Norcross schools  |  January 30, 2011 at 3:37 pm

    Fascinating! At least for now. That apparently might change. Thanks for the article and I LOVED Alan’s comment.

  • 3. Mark S  |  January 30, 2011 at 5:04 pm

    I think Alan is right here. The issue seems to be with acceptable confidence intervals. Currently, for whatever reason, the acceptable confidence interval for a bridge seems to be much larger than for psychotropic drugs. This seems to be an issue of perceived risk. People falling off bridges make a terrific spectacle for newspapers and the evening news, also the cause of the accident is clear. If someone commits suicide after taking a drug, how do we find out what the reason for that, especially if they were depressed to begin with? There are other cultural stigmas involved as well.

    I might also argue that pychortropic drugs making people feel better is a very fuzzy thing to measure vs. whether or not a bridge fell down or not. There is no placebo effect or mind science involved in the latter. Until we start counting neurons and watching them fire through a special microscope, psychology remains a very fuzzy research area.

    There are some other issues in the mix here it seems, too. Capitalism involves payment for studies as well. I have a Geologist friend who inspects concrete piers for buildings. He is hired on as a result of government regulations, but frequently encouraged by contractors and his own boss to be a rubber stamp instead of a true scientist or whistle-blower, because otherwise his firm is likely to be out of work.

    What labs are drug companies most likely to hire: ones which approve or disprove of whatever they’re marketing?

    It’s unfortunate, but these are the times we live in.

    • 4. Erik Engbrecht  |  January 30, 2011 at 7:55 pm

      What you’re saying doesn’t have anything to do with capitalism. It has to do with mixed incentives due to a lack of capitalism and an abdication of responsibility by the government.

      So the government doesn’t want buildings to fall down because it’s a public safety hazard. Builders don’t want their buildings to fall down because it subjects them to lawsuits and makes it unlikely that they will receive future business. But sometimes builders may be less risk averse than the government would like, so it decides to impose some regulations. In other words, it takes some of the land owners rights away regarding building by requiring some sort of approval.

      However, instead of hiring thoroughly trained and unbiased inspectors to ensure public safety needs are met, regulators license private contractors to do the assessments.

      So now the government has taken away from property rights and it has dictated who the builder can and cannot contract with to assess whether a site is suitable for building or not, and what kind of pilings are required to ensure that the building won’t fall down. This is not capitalism. It’s a controlled market created by the regulator.

      This is where the mixed incentives come in. Under a capitalist system, the builder would want as accurate of an assessment as possible of the site and what is required to build on it, so that he could balance cost and risk. But now an accurate assessment may completely close off certain options because the regulator won’t allow them, so now the builder has to balance between having full knowledge and risking losing the ability to build. So the builder often wants the stamp, so his options aren’t cut off. He can obtain that stamp, because licensed companies that frequently fail to provide it go out of business.

      By taking two whacks against capitalism instead of none or just one, the regulator turns clear and well aligned incentives into ones that are mixed and encourage deception.

      The problem is that there are often substantial incentives to control perception, because perception is what determines funding. Alleviating the problem requires first that incentives be clear, otherwise confusion alone is enough to create problems, so processes should be as simple as possible and not spread authority around needlessly and create circular dependencies among parties (e.g. the builder is dependent on the inspection company for approval, the inspection company is dependent on the builder for business). Second, the incentives must be aligned. I have no idea how to do this in the case of most research, because the researcher is always going to be incentivized to obtain funding, while funding providers need to optimize their allocations, which means denying some researchers funding. It doesn’t matter whether it’s in private industry, academia, government, or some hybrid.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,052 other followers


Recent Posts

Blog Stats

  • 2,030,788 hits
January 2011

CS Teaching Tips

%d bloggers like this: