Cry of Alarm

February 16, 2017

(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)

Stan Liebowitz writes:

Is it possible to respond to a paper that you are not allowed to discuss?

The question above relates to some unusual behavior from a journal editor. As background, I [Liebowitz] have been engaged in a long running dispute regarding the analysis contained in an influential paper published in one of the top three economics journals, the Journal of Political Economy, in 2007. That paper was written by Harvard’s Felix Oberholzer-Gee and Kansas’s Koleman Strumpf and appeared to demonstrate that piracy did not reduce record sales. I have been suspicious of that paper for many reasons. Partly because the authors publicly claimed that they would make their download data public, but never did, and four years later they told reporters they had signed a non-disclosure agreement (while refusing to provide the agreement to those reporters). Partly because Oberholzer-Gee is a coauthor with (the admitted self-plagiarist) Bruno Frey of two virtually identical papers (one in the JPE and one in the AER) that do not cite one another. But mostly because OS have made claims that they either knew were false, or should have known were false.

Although I have been critical of OS (2007) since its publication, it was not until September of 2016 that I published a critique in one of the few economics journals willing to publish comments and replications, Econ Journal Watch (EJW). [I also have a replication of a portion of their paper not reliant on their download data, that is currently under review at a different journal.] The editors of EJW invited Oberholzer-Gee and Strumpf (OS) to submit a response to my critique, to be published concurrently with my critique, but OS instead published their defense in a different journal,Information Economics and Policy (IEP, an Elsevier journal behind a paywall).

OS’s choice of IEP was not surprising. Among other factors, the editor of the journal, Lisa George, was a student of Oberholzer-Gee (he served on her dissertation committee), had coauthored two papers with him, and listed him as one of four references on her CV. IEP clearly fast-tracked the OS paper—it was first submitted to the journal on October 13, and the final draft, dated October 26, thanked three referees and the editor. The paper was published in December, although it often takes over a year from submission to publication in IEP.[1]

I had spent years attempting to get OS to publicly answer questions about their paper, so I was delighted that OS finally publicly defended their paper. Their published defense still left many questions unanswered, however, such as why the reported mean value of their key instrument was four times as large as its true value, but at least OS were now on the record, trying to explain some of their questionable data and results.

As a critic of their work, I took their published defense as a vindication of my concerns. Although their defense was superficially plausible, and was voiced in a confident tone, it was chock full of errors. For example, in EJW I had noted that OS’s data on piracy, which was the main novelty of their analysis, exhibited unusual temporal variability. I knew that OS might claim that this variability was a byproduct from a process of matching their raw piracy data to data on album sales, so I measured the variability of their raw piracy data prior to the matching process, and included a paragraph in EJW explicitly noting that fact. Yet in IEP, OS mischaracterized my analysis and claimed that the surprisingly large temporal variability was due to the matching process. Not only was their claim about my analysis misleading, but their assertion that the matching process could have materially influenced the variability of their data was also incorrect, which was clearly revealed by visual inspection of the data and a correlation of 0.97 between the matched and unmatched series. The icing on the cake was their attempt to demonstrate the validity of their temporal data by claiming a +0.49 correlation of their weekly data with another data set they considered to be unusually reliable. In fact, the correct correlation between those data sets was ‑0.68 (my rejoinder provides the calculations, raw data, and copies of the web pages from which the data were taken). All these errors were found in just the first section of their paper, with later sections continuing in the same vein.

After I became aware of the OS paper in IEP, I contacted the IEP editor and complained that I had not been extended the courtesy of defending my article against their criticisms. Professor George seemed to understand that fair play would require at least the belated pretense of allowing me to provide a rejoinder:

I welcome a submission from you responding to the Oberholzer – Strumpf paper and indeed intended to contact you about this myself in the coming weeks.

She also seemed to be trying to inflate the impact factor of her journal:

As you might be aware, IEP contributors and readers have rather deep expertise in the area of piracy. I would thus [ask] that in your response you take care to cite relevant publications from the journal. I have found that taking care with the literature review makes the referee process proceed more smoothly.

The errors made by OS in IEP seemed so severe that I thought it likely that IEP would try to delay or reject my submission, both to protect OS and to protect the reputation of IEP’s refereeing process. Still, I had trouble envisioning the reasons IEP might give if it decided to reject my paper.  I decided, therefore, to submit my rejoinder to IEP but to avoid a decision dragging on for months or years, I emphatically told Professor George that I expected a quick decision, and I planned to withdraw the submission if I hadn’t heard within two months.

Wondering what grounds IEP might use to reject my paper indicated an apparent lack of imagination on my part. Although the referees did not find any errors in my paper, the editor told me that she was no longer interested in “continued debate on this one paper [OS, 2007]” and that such debate was “not helpful to researchers actively working in this area, or to IEP readers.” Apparently one side of the debate was useful to her readers in December, when she published the OS article, but that utility had presumably evaporated by January when it came to presenting the other side of the debate.

Since Professor George was supposedly planning to “invite” me to respond to OS’s article, she apparently feels the need to keep up that charade, and does so by redefining the meaning of the word “response.” She stated: “I want to emphasize that in rejecting your submission I did not shut the door on a response…IEP would welcome a new submission from you on the topic of piracy that introduces new evidence or connects existing research in novel ways.”

Apparently, I can provide a “response,” but I am not allowed to discuss the paper to which I am supposedly responding. That appears to be a rather Orwellian request.

I have complained to Elsevier about the incestuous and biased editorial process apparently afflicting IEP. We will see what comes of it. The bigger issue is the quality of the original OS article, the validity of which seems even more questionable than before, given the authors’ apparent inability to defend their analysis. This story is not yet over.

  1.   The other papers in that issue were first received March 2014, September 2015, February 2015, November 2015, and April 2016.

Wow. We earlier heard from Stan Liebowitz on economics and ethics here and here. The above story sounds particularly horrible but of course we’re only hearing one side of it here. So if any of the others involved in this episode (I guess that would be Oberholzer, Strumpf, or George) have anything to add, they should feel free to so so in the comments, or they could contact me directly.

P.S. I hope everyone’s liking the new blog titles. I’ve so far used the first five on the list. They work pretty well.

The post Cry of Alarm appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

Tags: ,