Kara Weisman writes:
I’m a PhD student in psychology, and I attended your talk at the Stanford Graduate School of Business earlier this year. I’m writing to ask you about something I remember you discussing at that talk: The possible role of qualitative methods in addressing issues of replicability, reproducibility, and rigor.
In particular, I think you spoke about how many (all?) of us “quantitative” researchers in fact draw on more qualitative observations in piloting studies, forming hypotheses, etc., without labeling it as such. You asked us to consider how many people we really think we need to talk to in order to have a sense of a phenomenon (on the idea that’s it’s probably more like 3 or 4 rather than 300 or 400). I think part of the point here was that our statistical models don’t reflect how we actually think about the knowledge we gain from conducting experiments. (Forgive me if I’m mangling your ideas!)
Anyway, this got me thinking about the possible role of qualitative research and “mixed methods” in pushing psychology and other fields toward more replicable, reproducible, and rigorous research – the basic idea being that qualitative methods, though less “objective” than quantitative approaches, might have complementary properties that would be useful in forming priors/hypotheses, providing a check on what kinds of results seem reasonable, and guiding the interpretation and generalization of results.
Yes, I think these ideas are important, and I have not thought too systematically about it. This paper is somewhat relevant. Also you could look at this talk, where I discuss how we could possibly make progress in a world with weak theory and incremental improvements:
I agree with you 100% that qualitative methods are important. One way to think about this is that we learn through measurement, and qualitative research is used to decide what to measure and how to measure it. The other way to think about this is . . . where do treatments come from? Researchers just think them up, right? It presumably would be better to do this thinking-them-up more systematically, no? That’s qualitative research.
Qualitative research could guide what we measure (and how), as well as what we manipulate (and how). I think it could also guide what “priors” we bring to our analyses (formally or informally) and therefore what we make of our results. (And I’d argue that in some sense it already does – it’s just that most of us use informal intuitions from piloting/observation rather than systematic qualitative approaches, and then we don’t report these observations anywhere.)