Method Madness

Per this whole controversy about the Upward Bound evaluation, AERA’s always fabulous newsletter about what is happening in the research world reports that that there is an amendment in the education spending bill in the Senate to prescribe the methodology for the evaluation and shift it away from a randomized design. AERA is not pleased about it, and rightly so, this opens up a whole can of worms. Do we really want legislators deciding on the methodology for evaluating programs that they may — or may not — support? Why not just use the most optimal design?

2 Replies to “Method Madness”

  1. No, not optimal design. Researchers should use the most valid design. The design with the most scientific power to reveal program effects. It shouldn’t even be up for debate, actually.

    Randomized trials are the most powerful research design. Followed by observational studies that use methods such as propensity modeling to improve the chance to for causal claims.

    This is not some arbitrary conclusion that Congress can change with a wave of a wand.

  2. FYI, there is a now an amendment attached to the HEA reauthorization that cancels both the Upward Bound evaluation and the Absolute Priority that justifies it, AND says “The Secretary shall not require an eligible entity desiring to receive assistance under this chapter to recruit students to serve as a control group for purposes of evaluating any program or project assisted under this chapter.” Very disturbing precedent!

Leave a Reply

Your email address will not be published.