What should this student do? His bosses want him to p-hack and they don’t even know it!

November 11, 2017

(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)

Someone writes:

I’m currently a PhD student in the social sciences department of a university. I recently got involved with a group of professors working on a project which involved some costly data-collection. None of them have any real statistical prowess, so they came to me to perform their analyses, which I was happy to do. The problem? They want me to p-hack it, and they don’t even know it.

The project reads like one of your blog posts. The professors want to send this to a high-impact journal (they said Science, Nature, and The Lancet were their first three). There is no research question, and very little underlying theory. They essentially dumped the data on me and told me to email them when “you find something significant.” The worst part is, there is no malicious intent here and I don’t think they even know they they’re just fishing for p <.05. These are genuinely good, smart people who just want to do a cool study and get some recognition. I don't know if you have any advice to handling this sort of situation.

My recommendation is to do the best analysis you can, given your time constraints. If there are many potential things to look at, you might want to fit a multilevel model.

In any case, write up what you did, make graphs of data and fitted model, give the manuscript to the professors and let them decide where to submit it.

You’ll have a lot more control over the project if you write up your findings as a real paper, with a title, abstract, paragraphs, data and methods section, results, conclusions, and graphs. Don’t just send them a bunch of printouts as if you’re some kind of cog in the machine. Write something up.

My guess is that your colleagues/supervisors will appreciate this: Writing up results is a lot of work, and a student who can write is valuable. Here are some tips on writing research articles.

It’s fine if these profs want to change your paper, or rewrite it, or incorporate it into what you wrote (as long as they give you appropriate coauthorship). If in all this manipulation they want to submit something you don’t like, for example if they start pulling out p-values and telling bogus stories, then tell them you’re not happy with this! Explain your problems forthrightly. Ultimately it might come to a breakup, but give these colleagues of yours a chance to do things right, and give yourself a chance to make a contribution. And if it doesn’t work out, walk away: at least you got some practice with data analysis and writing.

The post What should this student do? His bosses want him to p-hack and they don’t even know it! appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

Tags: , , ,