Yes on design analysis, No on “power,” No on sample size calculations

Kevin Lewis points us to this paper, “Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty,” by Samantha Anderson, Ken Kelley, and Scott Maxwell.

My reaction: Yes, it’s reasonable, but I have two big problems with the general approach:

1. I don’t like talk of power because that’s all about trying to get statistical significance which I think is misguided. I think design analysis is important, and I think what Anderson et al. are doing is basically sound, but I just don’t think anyone should care what is the probability of getting “p less than 0.05” in a study. Yes, I know the idea is that the low p-value is a success and can be published, but I think this attitude is a mistake, and in my recent empirical work I’ve been firmly insisting that we not treat a low p-value (or a posterior interval excluding zero) as a determinant of success.

2. I think the focus should be on better measurements, not higher sample size. The question, “How large does N need to be?”, is a bit of a trap, in that if your measurements are really noisy, I don’t know that it’s such a good idea to try to sweep away your problems with a large sample size. Increasing N is a very crude way of decreasing variance, and it doesn’t do anything about bias at all. I say this in full awareness that Jennifer and I have a chapter on sample size calculations in our book. We have it in there because people feel the need to do it, but it bothers me.

My point in raising these issues is not to slam these three hardworking researchers who are making progress within the standard paradigm. We move forward one step at a time, after all. Their paper is fine. But I want to register my suggestion that we move forward, that we reduce the level of methodological research devoted to understanding and modeling epicycles, and move to a more direct or Copernican approach toward our study of statistics and research methods.