Fritz Strack sends along this article from 2012 which has an interesting perspective. Strack’s article begins: But, he continues, things changed in 2011 with the scandals of Diederik Stapel (a career built upon fake data), Daryl Bem (joke science getting published in a real journal), and a seemingly unending series of prominent studies that failed […]
Category: Sociology
“Troubling Trends in Machine Learning Scholarship”
Garuav Sood writes: You had expressed slight frustration with some ML/CS papers that read more like advertisements than anything else. The attached paper by Zachary Lipton and Jacob Steinhardt flags four reasonable concerns in modern ML papers: Recent progress in machine learning comes despite frequent departures from these ideals. In this paper, we focus on […]
Glenn Shafer tells us about the origins of “statistical significance”.
Shafer writes: It turns out that Francis Edgeworth, who introduced “significant” in statistics, and Karl Pearson, who popularized it in statistics, used it differently than we do. For Edgeworth and Pearson, “being significant” meant “signifying”. An observed difference was significant if it signified a real difference, and you needed a very small p-value to be […]
Junk science and fake news: Similarities and differences
Jingyi Kenneth Tay writes: As I read your recent post, “How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions” . . . and still stays around even after it’s been retracted, I realized that there are many similarities between this and fake news: how it is much easier to put fake news out […]
Funny citation-year thing
So, I’m going through the final draft of Regression and Other Stories, adding index entries, cleaning up references, etc., and I noticed this: Yes, I cite myself a lot—sometimes people call it “self-citation” and act like it’s a bad thing—but I think it’s helpful to point people to my earlier writings on various topics. Anyway, […]
They misreport their experiments and don’t fess up when they’ve been caught.
Javier Benitez points us to this paper, “COMPare: Qualitative analysis of researchers’ responses to critical correspondence on a cohort of 58 misreported trials,” by Ben Goldacre, Henry Drysdale, Cicely Marston, Kamal Mahtani, Aaron Dale, Ioan Milosevic, Eirion Slade, Philip Hartley and Carl Heneghan, who write: Discrepancies between pre-specified and reported outcomes are an important and […]
The State of the Art
Jesse Singal writes: This was presented, in Jennifer Eberhardt’s book Biased, as evidence to support the idea that even positive portrayals of black characters could be spreading and exacerbating unconscious antiblack bias. I did not see evidence to support that idea. I replied: I don’t understand what you’re saying here. I clicked thru and the […]
“Suppose that you work in a restaurant…”
In relation to yesterday’s post on Monty Hall, Josh Miller sends along this paper coauthored with the ubiquitous Adam Sanjurjo, “A Bridge from Monty Hall to the Hot Hand: The Principle of Restricted Choice,” which begins: Suppose that you work in a restaurant where two regular customers, Ann and Bob, are equally likely to come […]
“Statistical Inference Enables Bad Science; Statistical Thinking Enables Good Science”
As promised, let’s continue yesterday’s discussion of Christopher Tong’s article, “Statistical Inference Enables Bad Science; Statistical Thinking Enables Good Science.” First, the title, which makes an excellent point. It can be valuable to think about measurement, comparison, and variation, even if commonly-used statistical methods can mislead. This reminds me of the idea in decision analysis […]
Harking, Sharking, Tharking
Bert Gunter writes: You may already have seen this [“Harking, Sharking, and Tharking: Making the Case for Post Hoc Analysis of Scientific Data,” John Hollenbeck, Patrick Wright]. It discusses many of the same themes that you and others have highlighted in the special American Statistician issue and elsewhere, but does so from a slightly different […]
“Superior: The Return of Race Science,” by Angela Saini
“People so much wanted the story to be true . . . that they couldn’t look past it to more mundane explanations.” – Angela Saini, Superior. I happened to be reading this book around the same time as I attended the Metascience conference, which was motivated by the realization during the past decade or so […]
Things I didn’t have time to talk about yesterday at the Metascience conference
– Sane, reasonable, . . . and wrong. Here I was going to talk about some reasonable and moderate-sounding recommendations that I think miss some key issues. – The fractal nature of scientific revolution. I’ve talked about this a lot, for example in 2005, 2007, and 2012. – Workflow. My plan here was to talk […]
A world of Wansinks in medical research: “So I guess what I’m trying to get at is I wonder how common it is for clinicians to rely on med students to do their data analysis for them, and how often this work then gets published”
In the context of a conversation regarding sloppy research practices, Jordan Anaya writes: It reminds me of my friends in residency. Basically, while they were med students for some reason clinicians decided to get them to analyze data in their spare time. I’m not saying my friends are stupid, but they have no stats or […]
It’s not just p=0.048 vs. p=0.052
Peter Dorman points to this post on statistical significance and p-values by Timothy Taylor, editor of the Journal of Economic Perspectives, a highly influential publication of the American Economic Association. I have some problems with what Taylor writes, but for now I’ll just take it as representing a certain view, the perspective of a thoughtful […]
The ladder of social science reasoning, 4 statements in increasing order of generality, or Why didn’t they say they were sorry when it turned out they’d messed up?
Is the effect they found too large to believe? (the effect of breakfast micronutrients on social decisions)
Someone who wishes to remain anonymous writes: Have you seen this paper? I [my correspondent] don’t see any obvious problems, but the results fall into the typical social psychology case “unbelievably large effects of small manipulations”. They even say so themselves: We provided converging evidence from two studies showing that a relatively small variation in […]
“I am a writer for our school newspaper, the BHS Blueprint, and I am writing an article about our school’s new growth mindset initiative.”
Caleb VanArragon writes: I am a student at Blaine High School in Blaine, Minnesota. I am a writer for our school newspaper, the BHS Blueprint, and I am writing an article about our school’s new growth mindset initiative. I was wondering if you would be willing to answer a couple of questions about your study […]
“No, cardiac arrests are not more common on Monday mornings, study finds”
Paul Alper points us to this news report by Susan Perry. I have no idea how good this study is—I have not looked at it at all, except to pull out those two ugly-but-somewhat-functional graphs above (where “SCA” stands for “sudden cardiac arrest”)—but I wanted to convey my approval for a news story reporting a […]
Multilevel structured (regression) and post-stratification
My enemies are all too familiar. They’re the ones who used to call me friend – Jawbreaker Well I am back from Australia where I gave a whole pile of talks and drank more coffee than is probably a good idea. So I’m pretty jetlagged and I’m supposed to be writing my tenure packet, so […]
Amending Conquest’s Law to account for selection bias
Robert Conquest was a historian who published critical studies of the Soviet Union and whose famous “First Law” is, “Everybody is reactionary on subjects he knows about.” I did some searching on the internet, and the most authoritative source seems to be this quote from Conquest’s friend Kingsley Amis: Further search led to this elaboration […]