Tuesday, December 01, 2009

The Scientific Method

Being a remedial primer for AGW proponents Mann, Jones, et al. and a general introduction for the interested lay public ...

Your Humble Snarkatron is a scientist. This means, in addition to a visceral fondness for blinkenlights and tricks with liquid nitrogen, I conducted scientific research on a topic, presented papers at technical conferences, submitted papers (and had them published) at peer-reviewed, highly-regarded journals in the field of solid state physics, and also that I put together an independent body of work and defended it before a committee of my scientific peers to earn my PhD. This thesis is publicly available, as are the papers and data I published. The mere thought of concealing my experimental data makes me writhe in intellectual agony. Dreaming about losing my data would make me wake up in a cold sweat. I STILL HAVE THE ORIGINAL DATA FILES, over 15 years later.

This is the way the system works.
-Researcher A makes some measurements, using a specified technique, sample prep procedure, etc. These measurements and the experimental configuration are recorded in some permanent form.
-Researcher A, or more likely Researcher A's grad students, graph the data in a useful form and write up a paper explaining why this is such a big deal for the scientific community in general and the sub-community interested in sample/measurement technique in particular.
-Paper is submitted to Journal of Interesting Results. Editor of JIR looks through rolodex to find a reviewer *not* on the list of authors of the paper but who is familiar with sample/measurement technique. Paper is sent out to victim^H^H^H reviewer with proper groveling note giving thanks in advance for reading and commenting.
-Researcher B gets paper and reads with proper scientific detachment, making sure all assertions are supported with data, that the data makes reasonable sense given their experience with material/measurement technique, and sometimes even correcting spelling. NOTE: Researcher B should not be a tennis buddy, sexual partner, relative, or otherwise owe Researcher A money. It is not okay for Researchers A & B to take turns wearing the French Maid costume so the other can pretend not to know them.
-Researcher B returns paper with comments along with a go/no-go recommendation. The comments and recommendation (but NOT the name and affiliation of the reviewer) are forwarded to Researcher A, who can contest an unfavorable review if one is given.
-Once a favorable review is obtained, the editor of the Journal of Interesting Results schedules the paper for publication. The favorable review, I will point out, may NOT come from Researcher A, nor may the editor "shop around" reviewers to get a favorable review.
-The paper, now published, is Fair Game. Other researchers in the field, perusing the Journal of Interesting Results over a cup of Ovaltine, may be moved to try and reproduce the interesting result in question, possibly adding a few additional experimental wrinkles to obtain more insight. If they cannot reproduce the result, Things Happen. Letters are written to Researcher A, and possibly the editor of JIR. Clarification may ensue, or profound academic embarrassment when Researcher A realizes the measurements were in error and the Journal of Interesting Results has to let everyone know to ignore the previous paper (otherwise known as a "retraction"). Much, much more rarely, Researcher A cannot or will not produce a valid explanation for why a sample would ever measure -10,000 ohms, and then we move into the interesting area known as Academic Dishonesty.

I have submitted multiple papers for review, have acted as a reviewer for a very large and well-known physics journal, and have even (successfully) contested an unfavorable review. This stuff is not complicated. Time-consuming, but not complicated. It never crossed my fevered brain to say a researcher could not view my published data on the grounds "they will just try to find something wrong with it." Of COURSE they are going to try and find something wrong with it, that's what this process is about! Publishing a scientific paper is not like handing your latest fingerpainting to your doting mother to post on the refrigerator door. The correct attitude is "do your damndest. I hope you choke on my excellent work and well-supported arguments--and why don't you invite your friends?"

The gang involved with the recent Climategate leaked email scandal (or Climatequiddick, my favorite) were climate researchers. Such *can* be scientists, with all that implies, but they labor under some important difficulties. For one thing, we've only got the one working planet at the moment, and My Little Solar Accretion Disk kits are not yet on the market. I.e. we only have one experimental subject. Further, our ancestors were very inconsiderate and did not keep records of temperature, rainfall, etc. at the level of accuracy we demand to detect .05 C global temperature changes caused by my penchant for bonfires. Sometimes the blighters didn't keep records at all, their excuse being they hadn't invented writing yet. So, much use is made of temperature proxies, such as the infamous tree ring measurements. Trees, I should remind you, do not have thermometers. They tend to grow when they are not freezing, but that's about it. They get more excited about availability of water. A big fat ring *could* mean a temperate year, but with lots of precipitation. Oh, and really old trees tend to be in far off, inaccessible-to-lumberjacks type areas. Hence, these data sets of temperature proxies aren't something you can order up on Amazon. A research team that has gone to the trouble and expense of blackmailing a grad student into spending a summer taking core samples in Siberia *without beer* would, naturally, want first dibs at this data. First dibs is entirely understandable, a 25 year embargo is a scandal.

So, we have 1) an experimental system so huge and a timescale so vast it is incredibly difficult to amass a useful set of data to confirm or deny climate research claims, 2) a rather small (by scientific standards) group of researchers in the field, pretty much guaranteeing a high percentage of incestuous research relationships and therefore a reviewer pool with a significant chance of being "improperly interested" in the paper under review, and 3) modeling that conveniently predicted measurable results in the global temperature increase area well after everybody important had retired. This makes any claim of what *I* understand as "peer review" impossible. If there isn't a very good chance sloppy work will get pointed to and laughed at during Researcher A's academic career, why even bother?


Anonymous BillT said...

Trees are useful as thermometers in this part of the world. We know that Spring has officially arrived when they spontaneously combust.

We know when *Summer* has arrived when small arthropods spontaneously combust...

9:46 PM, December 01, 2009  
Anonymous BillT said...

BTW, here's a present for you:


9:48 PM, December 01, 2009  
Anonymous wolfwalker said...

Question, BCR: When the manuscript of a paper gets sent out to Researcher B, is it stripped of author information? I have the impression from somewhere that just as the author is not allowed to know the names of reviewers, the reviewers are not allowed to know the names of the authors. Just in case Researcher B has a bias either way concerning Researcher A.

5:00 AM, December 02, 2009  
Blogger Carolyn said...

I am but a humble Padawan Learner, and am debating whether or not to continue on..I am not sure I could face the Trials of a peer review or the editor's white out.

That said, I have kept every single bleeping paper I have written JIC. I have underlined lines of text in my Tomes to Prove a Point in case I needed to defend an answer.

And I am not even a scientist or an acada-thingy, but a mere College Student.

6:47 AM, December 02, 2009  
Blogger Snarkatron said...

Bill, you are confusing thermometers with calendars. I know, you think sundials are newfangled technology. Try to keep up. Thanks for Clippy -- isn't he a helpful little avatar?

wolfwalker, in my field we always had the name(s) of the author and affiliation (the university or laboratory they were with) for a reviewed paper. The secret would be out in any case once the paper is published.

8:05 AM, December 02, 2009  
Blogger The Thomas said...

Worse yet those weather people only recalibrate their thermometers when they are known to be 3 degrees out of cal.

That puts a rather large error bar on the data. With +/- 1.5 degress C as your error bar, differences under 1.5 degrees are noise and essentually zero.

With all the differences in the noise, you can not have a resolved measurement of 0.5 degrees change. The data is not there.

Some weather people have made it worse by moving their sampling stations from the required acre of grass into the middle of black-top (and so black body) parking lots, or next to air conditioning units (which keep the thermometers nice and warm).

WV: traters (umm, baked traters)

8:25 AM, December 02, 2009  
Anonymous BillT said...

Bill, you are confusing thermometers with calendars.

Am *not*. Calendars are the things with pictures at the top and your insurance agent's phone number on the bottom.

The Thomas: Even more fun arises when the temps are "adjusted" because the sensor suites were moved a couple hundred meters uphill and they adjust using the standard adiabatic lapse rate rather than the dry or wet, depending on location. And *still* fark the adjustment factor because they convert meters to feet and then round up to the nearest *whole* metric equivalent...

12:21 AM, December 03, 2009  
Anonymous BillT said...

How're your Mad Coding Skillz holding out?

You'll *love* this one:


6:58 AM, December 04, 2009  

Post a Comment

<< Home