Randomized Controlled Trials Part II: The Scientific Method

Remember the science projects of your childhood? Thinking about wielding the bi-fold boards onto the kitchen table, cutting colored paper, and pasting the various hypotheses, data, and results make me nostalgic. This trip down memory lane occurs whenever I consider the scientific method because these memories are familiar. My science fair projects were never anything special; however, even now they seem illustrative. For example, do plants grow better in the presence of classical music, rock music, or no music? So, my dependent variable was plant growth. The plants were identical in every respect. Same bag of seeds. Same soil. Same sunlight. Same watering. The only difference was the exposure to different music. My control group was "no music". My treatment groups were "rock music" and "classical music". Perhaps the question was juvenile and the sample size small, but, when you think about it, this is the scientific method at its best.

For a moment, consider a question that personally interests you. Will school uniforms improve educational outcomes? Does religious proscription improve generosity? Does merit-based teacher pay improve student educational outcomes? What will the impact of a different tax structure do to labor supply? What is the impact of job training programs on ex-convicts? Does marriage make people happier? And, the list could go on, and on, and on.

The scientific gold standard is a "randomized controlled trial" with a large sample size. The term "randomized controlled trial" is borrowed from the medical and clinical fields and is quite descriptive. Randomization matters because it reduces self-selection bias.

For example, imagine that we were studying the impact of a government sponsored job training programs for ex-convicts. The before-and-after study may show tremendous success in transitioning ex-convicts into the workforce; however, substantial information would be missing. If the job training program is voluntary there could be differences in motivation, education, etc. amongst some of the ex-convicts compared to their counterparts that did not sign up. Therefore, the question would be, "How much is the job training helping, or, would these people have found work regardless of the program simply because they are highly motivated to turn their life around?" 

Randomization can help overcome the selection bias. Randomization would assign people into job training programs and thus weed out any underlying differences between the two sets of populations (those who would sign up and those who would not sign up).

Controlled implies that the only difference on average between people in the sample will be whether or not they received job training. Like my flowers example where the only difference was exposure to different music, the only difference, on average, among those being released from jail was whether they received job training or not. Certainly there will still be differences between people released who receive job training and those who do not; however, those differences will be balanced.

Obviously there are some questions that cannot be answered by randomized controlled trials. For example, the Human Subjects Committee wouldn't allow me to randomly assign people to religions in order to observe how certain religious teachings alter generosity. Also, they would likely disallow random assignment of people into marriages to figure out whether marriage causes them to become happier. These are good things! But, there are a number of really interesting questions that can be answered with randomized controlled trials. In my next post I will discuss some of the interesting "field experiments" (equivalent term for RCT) that have been conducted in the past and more recently.

By the way, the plants grew better under classical music. Though this is far from an iron-clad empirical reality.