Did that “bottomless soup bowl” experiment ever happen?

I’m trying to figure out if Brian “Pizzagate” Wansink’s famous “bottomless soup bowl” experiment really happened.

Way back when, everybody thought the experiment was real. After all, it was described in a peer-reviewed journal article.

Here’s my friend Seth Roberts in 2006:

An experiment in which people eat soup from a bottomless bowl? Classic! Or mythological: American Sisyphus. It really happened.

And here’s econ professor Richard Thaler and law professor Cass Sunstein in 2008:

Given that they described this experiment as a “masterpiece,” I assume they thought it was real.

Evidence that the experiment never happened

I’ve known for awhile that some of the numbers in the Wansink et al. “Bottomless bowls” article were fabricated, or altered, or mis-typed, or mis-described, or something. Here’s James Heathers with lots of details.

But I’d just assumed there really had been such an experiment . . . until I encountered two recent blog comments by Jim and Mary expressing skepticism:

For me, for sure, if I got a 6oz soup bowl that refilled itself without me knowing I’d just go right on eating gallon after gallon of soup, never noticing. . . . There’s no way he even did that! That has to be a complete fabrication.

If you try to imagine designing the refilling soup bowl, it gets harder and harder the more you think about it. The soup has to be entering the bowl at exactly the right rate. . . . I don’t think they really did this experiment. They got as far as making the bowls and stuff, but then it was too hard to get it to work, and they gave up. This would explain why an experimental design with 2 bottomless and 2 non-bottomless subjects per table ended up with 23 controls and 31 manipulations . . .

I searched the internet and found a photo of the refilling soup bowl. Go to 2:36 at this video.

See also this video with actors (Cornell students, perhaps?) which purports to demonstrate how the bowl could be set up in a restaurant. The video is obviously fake so it doesn’t give me any sense of how they could’ve done it in real life.

I also found this video where Wansink demonstrates the refilling bowl. But this bowl, unlike the one in the previous demonstration, is attached to the table so I don’t see how it could ever be delivered to someone sitting at a restaurant.

So when you look at it that way: an absurdly complicated apparatus, videos that purport to be reconstructions but which lack plausibility, and no evidence of any real data . . . If seems that the whole thing could be a fake, that there was no experiment after all. Maybe they built the damn thing, tried it out on some real students, it didn’t work, and then they made up some summary statistics to put in the article. Or they did the experiment in some other way—for example, just giving some people more soup than others, with the experimentalists rationalizing it to themselves that this was essentially equivalent to that bottomless-bowl apparatus—and then fudged the data at the end to get statistically significant and publishable results.

Or maybe it all happened as described, and someone just mistyped a bunch of numbers which is why the values in the published paper didn’t add up.

To paraphrase Jordan Anaya: I dunno. If I’d just designed and carried out the most awesome experiment of my career—a design that some might call a “masterpiece”—I think I’d be pretty damn careful with the data that resulted. I’d’ve made something like 50 copies of the dataset to make sure it never got lost, and I’d triple-check all my analyses to make sure I didn’t make any mistakes. I might even bring in two trusted coauthors just to be 100% sure that there were no missteps. I wouldn’t want to ruin this masterpiece.

It’s as if Wansink had found some rare and expensive crystal goblet and then threw it in the back of a pickup truck to bring it home. A complete disconnect between the huge effort required to purportedly collect the data, and the zero or negative effort expended on making sure the data didn’t get garbled or destroyed.

Evidence that the experiment did happen

On the other hand . . .

Perhaps the strongest argument in favor of the experiment being real is that there were three authors on that published paper. So if the whole thing was mde up, it wouldn’t be just Brian Wansink doing the lying, it would also be James Painter and Jill North. That moves our speculation into the conspiracy category.

That said, we don’t know how the project was conducted. It might be that Wansink took responsibility for the data collection, and Painter and North were involved before and after and just took Wansink’s word for it that the experiment was actually done. Or maybe there is some other possibility.

Another piece of evidence in favor of the experiment being real is that Wansink and his colleagues put a lot of effort into explaining how the bowl worked. There are three paragraphs in Wansink et al. (2005) describing how they constructed the apparatus, how it worked, and how they operated it. Wansink also devotes a few pages of his book, Mindless Eating, to the soup experiment, providing further details; for example:

Our bottomless bowls failed to function during the first practice trial. The chicken noodle soup we were using either clogged the tubes or caused the soup to gurgle strangely. We bought 360 quarts of Campbells tomato soup, and started over.

I’m kinda surprised they ever thought the refilling bowl would work with chicken noodle soup—isn’t it obvious that it would clog the tube or clump in some way?—but, hey, dude’s a b-school professor, not a physicist, I guess we should cut him some slack.

Scrolling through the Mindless Eating on Amazon, I also came across this:

It seems that when estimating almost anything—such as weight, height, brightness, loudness, sweetness, and so on—we consistently underestimate things as they get larger. For instance, we’ll be fairly accurate at estimating the weight of a 2-pound rock but will grossly underestimate the weight of an 80-pound rock. . . .

They’re having people lift 80-pound rocks? That’s pretty heavy! I wonder what the experimental protocol for that is. (I guess they could ask people to estimate the weight of the rock by just looking at it, but that would be tough for lots of reasons.)

But I digress. To return to the soup experiment, Wansink also provides this story about one of the few people who had to be excluded from the data:

Cool story, huh? Not quite consistent with the published paper, which simply said that 54 participants were recruited for the study, but at least some recognition that moving the soup bowl could create a problem.

Summary

Did the experiment ever happen? I just don’t know! I see good arguments on both sides.

I can tell you one thing, though. Whether or not Wansink’s apparatus ever made its way out of the lab, it seems that the “bottomless soup bowl” has been used in at least one real experiment. I found this paper from 2012, Episodic Memory and Appetite Regulation in Humans, by Jeffrey Brunstrom et al., which explains:

Soup was added or removed from a transparent soup bowl using a peristaltic pump (see Figure 1). The soup bowl was presented in front of the volunteers and it was fixed to a table. A tall screen was positioned at the back of the table. This separated the participant from both the experimenter and a second table, supporting the pump and a soup reservoir. Throughout the experiment, the volunteers were unable to see beyond the screen.

The bottom of the soup bowl was connected to a length of temperature-insulated food-grade tubing. This connection was hidden from the participants using a tablecloth. The tubing fed through a hole in the table (immediately under the bowl) and connected to the pump and then to a reservoir of soup via a hole in the screen. The experimenter was able to manipulate the direction and rate of flow using an adjustable motor controller that was attached to the pump. The pre-heated soup was ‘creamed tomato soup’ (supplied by Sainsbury’s Supermarkets Ltd., London; 38 kcal/100 g).

And:

Participants were then taken to a testing booth where a bowl of soup was waiting. They were instructed to avoid touching the bowl and to eat until the volume of soup remaining matched a line on the side of the bowl. The line ensured that eating terminated with 100 ml of soup remaining, thereby obscuring the bottom of the bowl.

So it does seem like the bottomless soup bowl experiment is possible, if done carefully. The above-linked article by Brunstrum et al. seems completely real. If it’s a fake, it’s fooled me! If it’s real, and Wansink et al. (2005) was fake, then this is a fascinating case of a real-life replication of a nonexistent study. Kind of like if someone were to breed a unicorn.