(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Bob Reed responded to my recent Retraction Watch article (where I argued that corrections and retractions are not a feasible solution to the problem of flawed science, because there are so many fatally flawed papers out there and retraction or correction is such a long, drawn-out process) with a post on openness, data transparency, and replication. He writes:
A recent survey article by Duvendack et al. report that, of 333 journals categorized as “economics journals” by Thompson Reuter’s Journal Citation Reports, 27, or a little more than 8 percent, regularly published data and code to accompany empirical research studies. As some of these journals are exclusively theory journals, the effective rate is somewhat higher.
Noteworthy is that many of these journals only recently instituted a policy of publishing data and code. So while one can argue whether the glass is, say, 20 percent full or 80 percent empty, the fact is that the glass used to contain virtually nothing. That is progress.
But making data more “open” does not, by itself, address the problem of scientific unreliability. Researchers have to be motivated to go through these data, examine them carefully, and determine if they are sufficient to support the claims of the original study. Further, they need to have an avenue to publicize their findings in a way that informs the literature. . . .
Without an outlet to publish their findings, researchers will be unmotivated to spend substantial effort re-analysing other researchers’ data. Or to put it differently, the open science/data sharing movement only addresses the supply side of the scientific market. Unless the demand side is addressed, these efforts are unlikely to be successful in providing a solution to the problem of scientific unreliability.
Reed concludes:
The irony is this: The problem has been identified. There is a solution. The pieces are all there. But in the end, the gatekeepers of scientific findings, the journals, need to open up space to allow science to be self-correcting.
Could be. It could also be that the journals become less important. In statistics and political science, I see journals as still being very important in determining academic careers, but not so important any more for the dissemination of knowledge. There’s just so much being published and so many places to look for things. Econ might be different because there are still a few journals that are recognized to be the top. From the other direction, the CS publication system is completely broken because all the conferences are flooded with papers, it’s just endless layers of hype. CS, on the other hand, is doing just fine. At this point the publication process just seems to be going in parallel with the research process, not really doing anything very useful.
The post Transparency, replications, and publication appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
