Please contribute to this list of the top 10 do’s and don’ts for doing better science

October 10, 2017

(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)

Demis Glasford does research in social psychology and asks:

I was wondering if you had ever considered publishing a top ten ‘do’s/don’ts’ for those of us that are committed to doing better science, but don’t necessarily have the time to devote to all of these issues [of statistics and research methods].

Obviously, there is a lot of nuance in both methods and stats for any particular project. So, I’m not asking you for a ‘one size fits all’, but more of a 5 or 10 factor checklist as a framework for those of us committed to doing better work, but worried we may not have the expertise or time to follow-through on these commitments. Sort of a—whatever you do, at least do x, y, and z.

I looked up Glasford on the internet and found this description of his research:

The focus of much of my work is on three interrelated streams of research questions that are concerned with understanding: how people make decisions about what to do when faced with injustice; what compels people to join and stay involved in political protest that can benefit their own group, as well as groups they do not belong to; and when, why, and what helps individuals from groups of differing power improve relations with one another.

Wow—this sounds important. I should talk with this guy.

In the meantime, do I have a checklist of 10 items? I’ve given advice to psychology researchers from time to time but I don’t have a convenient list of 10 things.

But we should have such a list! Can you make some suggestions in comments? Also, if anyone out there is in contact with any leading social psychologists, maybe we could get their thoughts too? There’s a lot I disagree with in the writings of, say, Susan “terrorists” Fiske or Daniel “shameless little bullies” Gilbert or Mark “Evilicious” Hauser or all the other people you’re sick of hearing about on this blog—but, say what you want about these people, they’ve thought a lot about psychology research and I’d be interested in what their top 10 tips would be. Not tips on how to get published in PNAS or wherever, but tips on doing better science.

Unfortunately I don’t expect we’ll hear from the above people (I’d be happy to be surprised on that end, though!), so in the meantime I’d love to hear your thoughts.

OK, I’ll start with items #1, 2, and 3 on the top-10 list, in decreasing order of importance:

1. Learning from data involves three stages of extrapolation: from sample to population, from treatment group to control group, and from measurement to the underlying construct of interest. Worry about all three of these, but especially about measurement, as this tends to be taken for granted in statistics discussions.

2. Variance is as important as bias. To put it another way, take a look at your (estimated) bias and your standard error. Whichever of these is higher, that’s what you should be concerned about.

3. Measurement error and variation are concerns even if your estimate is more than 2 standard errors from zero. Indeed, if variation or measurement error are high, then you learn almost nothing from an estimate even if it happens to be “statistically significant.”

OK, none of the above are so pithy, and I’m open to the idea of other items bumping these down the list.

It’s your turn.

The post Please contribute to this list of the top 10 do’s and don’ts for doing better science appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

Tags: , ,