(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Robert Bloomfield writes:
Most of the people in my field (accounting, which is basically applied economics and finance, leavened with psychology and organizational behavior) use ‘positive research methods’, which are typically described as coming to the data with a predefined theory, and using hypothesis testing to accept or reject the theory’s predictions. But a substantial minority use ‘interpretive research methods’ (sometimes called qualitative methods, for those that call positive research ‘quantitative’). No one seems entirely happy with the definition of this method, but I’ve found it useful to think of it as an attempt to see the world through the eyes of your subjects, much as Jane Goodall lived with gorillas and tried to see the world through their eyes.)
Interpretive researchers often criticize positive researchers by noting that the latter don’t make the best use of their data, because they come to the data with a predetermined theory, and only test a narrow set of hypotheses to accept or reject. In contrast, the interpretive researchers dwell in their data (interviews and observation) to develop their theories. Sure, it might fail Popper’s requirements of falsifiability, but is that such an unreasonable tradeoff if you get to use all of your data?
Now to my questions. First, how different is the interpretive argument from the usual Bayesian critique of frequentist hypothesis testing? Isn’t much of the argument that frequentist hypothesis-testers are ignoring the bulk of their data in order to get a yay or nay on one narrow question? Second, does the increased focus on big data make quantitative statistics even more consistent with the interpretive view? If someone is given the first crack at a big data set, isn’t the Bayesian recommendation to wallow in the data and use it to develop theory, rather than spell out some narrow hypotheses and try to reject or accept them using (most likely) a small slice of the big data?
No doubt I’ve muddied the issues by using loaded terms like Bayesian, frequentist, and big data. But hopefully the question makes enough sense to be answered. Any references you or your readers could provide would be greatly appreciated.
As an aside, I should mention that I think that most positive researchers aren’t nearly as pure as Popper and other falsificationists would demand. Most of the studies in my field are statistical analyses of the exact same data sets studied by many researchers before them (stock prices, accounting disclosures and the like), so basically every paper tests theories that have been developed by others who examined almost exactly the same data sets. And no one admits it, but they all run a host of analyses before reporting their final hypothesis tests. I’m not criticizing, only noting that they seem to dwell in their data more like an interpretive researcher than one would guess from reading their published papers.
My reply:
In political science, I’ve sometimes heard the phrase, “empirical implications of theoretical models,” which sounds similar to what is called “positive research methods” in your field. In either case it looks like a sort of extreme Popperism that goes as follows:
1. Researcher A uses theory to come up with a clear set of testable hypotheses.
2. Researcher B gathers data and comes to a conclusion, which is either:
(a) Whew! The hypothesis is not rejected. It lives another day and is stronger by virtue of surviving the test; or
(b) Wow! The hypothesis is rejected. Start the revolution.
3. Either way, you win. You’ve either supported a model you like, or you’ve shot down a formerly-live model. Typically, though, the goal is (a). You’re supposed to specify the hypothesis in a way that can be testable, then you’re supposed to design the data collection in a way so that you’re likely to learn something about the hypothesis (or perhaps to distinguish between two hypotheses, although that formulation has always made me uncomfortable, in that once we entertain hypothesis X and hypothesis Y, I’d like to do continuous model expansion and consider X and Y to be special cases of a more general hypothesis Z, and at this point I consider “X or Y” to be a dead question.
In theory, researchers A and B can come from different research groups—they don’t even need to know each other—but there seems to be a sense that it’s best for them to be the same person. In this way, empirical implications of theoretical models seems to be following the James Taylor rather than the Frank Sinatra model. The singer is the songwriter.
One difficulty of the Empirical Implications paradigm is that, in practice, the testable hypotheses can be pretty vague and can map many different ways into empirical data. This is what Eric Loken and I call the garden of forking paths.
Popper turned upside down
Even setting aside any difficulties of specifying research hypotheses, one odd thing about the above Popperian approach is that the hypothesis being tested is, in the terms of classical statistical analysis, an “alternative hypothesis” not a “null hypothesis.”
To put it yet another way, the research hypothesis (the “theoretical model” whose “empirical implications” are being tested) is (typically) something you want to uphold, whereas the statistical null hypothesis is a straw man (or, as Dave Krantz would say, a straw person) that you want to reject.
And I’m not just playing with words here. The two hypotheses are exact hypotheses, in that support of the research hypothesis (an outcome of the form 2(a) in the above story) is typically defined precisely as a statistical rejection of the null hypothesis.
This is Popper turned upside down. It could still all make sense—but we just have to keep in mind that rejection in the Neyman-Pearson sense counts as non-rejection in the Popper sense, and non-rejection in the Neyman-Pearson sense counts as rejection in the Popper sense. I don’t think this point is generally clear at all.
To get back to the original question
In my own research, I think data exploration is super-important. Much of my most influential and important applied work never could or would have happened had I been required to follow the Empirical Implications playbook and write out my hypotheses ahead of time. For example, see here, here, and here. In all these cases, it was only through lots of data analyses that my collaborators and I came to our research hypotheses. I see the appeal of Empirical Implications in pushing researchers to a sort of intellectual discipline but I don’t think it would work for me.
Finally, I think the whole Bayesian thing is a red herring here. As Uri Simonsohn has explained, you can p-hack just as well with Bayes as with any other approach. As I see it, the problem is not with classical rules or even with p-values but rather comes earlier, with a confusion between research hypotheses and statistical hypotheses and an attitude that an extreme p-value or posterior probability or whatever can be taken as strong evidence, without reference to how that number came about.
The post Empirical implications of Empirical Implications of Theoretical Models appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
