The gaps between 1, 2, and 3 are just too large.

Someone who wishes to remain anonymous points to a new study of David Yeager et al. on educational mindset interventions (link from Alex Tabarrok) and asks:

On the blog we talk a lot about bad practice and what not to do. Might this be an example of how *to do* things? Or did they just get lucky? The theory does not seem any stronger than for myriad other too-noisy-to-say-anything studies.

My reply: Hey, I actually was involved in that project a bit! I don’t remember the details but I did help them in some way, I think at the design stage. I haven’t looked at all the details but they seemed to be doing all the right things, including careful measurements, connection to theory, and analysis of intermediate outcomes.

My correspondent also asks:

Also, if we need 65 random schools and 12,000 students to do a study, I fear that most researchers could not do research. Is it pointless to do small studies? I fear they are throwing out the baby with the bathwater.

My reply:

You don’t need such a large sample size if you collect enough data on each individual student and class. Don’t forget, you can learn from N=1 in a good qualitative study—after all, where do you think the ideas for all these interventions came from? I do think, though, that the sloppy-measurement, small-N study is not such a good idea: in that case, it’s all bathwater with no baby inside.

What we really need are bridges between the following three things:

1. Qualitative research, could be N=1 or could be larger N, but the point is to really understand what’s going on in individual cases.

2. Quantitative research with careful measurement, within-person comparisons, and large N.

3. The real world. Whatever people are doing when they’re not doing research.

The gaps between 1, 2, and 3 are just too large.

The post The gaps between 1, 2, and 3 are just too large. appeared first on Statistical Modeling, Causal Inference, and Social Science.