Is the effect they found too large to believe? (the effect of breakfast micronutrients on social decisions)

Someone who wishes to remain anonymous writes:

Have you seen this paper?

I [my correspondent] don’t see any obvious problems, but the results fall into the typical social psychology case “unbelievably large effects of small manipulations”. They even say so themselves:

We provided converging evidence from two studies showing that a relatively small variation in breakfast’s macronutrient composition has a striking impact on social decisions.

The article in question is “Impact of nutrition on social decision making,” by Sabrina Strang, Christina Hoeber, Olaf Uhl, Berthold Koletzko, Thomas F. Münte, Hendrik Lehnert, Raymond Dolang, Sebastian Schmid, and Soyoung Park. From the abstract:

Breakfasts with a high-carbohydrate/protein ratio increased social punishment behavior in response to norm violations compared with that in response to a low carbohydrate/ protein meal. We show that these macronutrient-induced behavioral changes in social decision making are causally related to a lowering of plasma tyrosine levels.

And here’s their evidence:

I’m concerned about implausible effect size estimates, which is what you can get from the combination of noisy data, small samples, and forking paths in analysis.

The graph on the left is from experiment 1 which is observational data (not assigning breakfasts but just asking people what they ate), but still:

Within the low-carb/ protein group, 24% of subjects decided to reject unfair offers. In contrast, 53% of the high-carb/protein group decided to reject unfair offers.

I don’t care if it is p=0.03, I don’t expect to see this in a replication.

There’s a lot more data, and there could be something going on—I have no idea. I think they should do a Nosek, Spies, and Motyl and replicate the whole thing from scratch. Or someone else can do the replication.

Until then, I’m skeptical of these claims:

The findings indicate that, in a limited sense, “we are what we eat” and provide a perspective on a nutrition-driven modulation of cognition. The findings have implications for education, economics, and public policy, and emphasize that the importance of a balanced diet may extend beyond the mere physical benefits of adequate nutrition.

Or this:

In this study, we demonstrated that the macronutrient composition of food acutely influences our social decisions, showing a modulation in the dopamine precursor as the underlying mechanism.

Exploratory experimentation and analysis are fine—that’s what science is all about. Let’s just not forget that finding some statistical significant comparisons in data is not the same thing as scientifically “demonstrating” a hypothesis. Their hypothesis could well be true, or maybe not, or maybe it depends on context. Nothing special about this particular study, we just need to give such studies a modern reading.