A reader who wishes to remain anonymous writes:
I would be curious to hear your thoughts on is motivated reasoning among open science advocates. In particular, I’ve noticed that papers arguing for open practices have seriously bad/nonexistent causal identification strategies.Examples:
Kidwell et al. 2017, Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency. Published in Plos Bio, criticized in Plos blog here. Brian Nosek responds at great length therein.McKiernan et al. 2017, Point of View: How open science helps researchers succeed, claims that there is “evidence that publishing openly is associated with higher citation rates,” but also notes that “some controlled studies have failed to find” an effect. A glance through the citations suggests that all the RCTs find null effects, and the observational studies find substantial effects.3) Rowhani-Farid and Barnett 2018, “Badges for sharing data and code at Biostatistics: an observational study” — compares citations for articles in a journal (biostatistics) that introduced badges to those in a journal that did not (statistics in medicine); the causal identification strategy is that the two journals are “in the same field of research with similar goals of publishing papers on statistical methods development in health and medicine.” The article finds that the “effect of badges at Biostatistics was a 7.3% increase in the data sharing rate.” (Further down, they write that their study “cannot accurately deduce the effectiveness of badges because of the biases of the non-randomised study design.” Well, then how should we interpret the claims in the abstract??)So, one thing that makes me feel sad about this is that they’re all published in journals with a clear stake in open access norms — PLOS, eLife, and F1000. I worry that publishing articles like these discredits the model.Also, I do think there are large, well-justified benefits to open science practices. Davis 2011 finds that “[a]rticles placed in the open access condition (n=712) received significantly more downloads and reached a broader audience within the first year, yet were cited no more frequently, nor earlier, than subscription-access control articles (n=2533) within 3 yr.” David Donoho (2017) writes that “[w]orking from the beginning with a plan for sharing code and data leads to higher quality work, and ensures that authors can access their own former work, and those of their co-authors, students and postdocs.” But I guess that there is still demand for research showing a strong citation benefit to open scholarship, regardless of what the evidence says.
I don’t have the energy to read these papers—I guess I don’t really care so much if open-science increases citation rates by 17% or whatever—but I agree with the general principle expressed by our correspondent that it’s not good practice to exaggerate evidence, even in a good cause.