(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Last year I came across an article, “Labor Market Returns to Early Childhood Stimulation: a 20-year Followup to an Experimental Intervention in Jamaica,” by Paul Gertler, James Heckman, Rodrigo Pinto, Arianna Zanolini, Christel Vermeerch, Susan Walker, Susan M. Chang, and Sally Grantham-McGregor, that claimed that early childhood stimulation raised adult earnings by 42%. At the time, I wrote,
Overall I have no reason to doubt the direction of the effect—psychosocial stimulation should be good, right?—but I’m skeptical of the 42% claim, for the usual reasons of the statistical significance filter. . . . There’s nothing wrong with speculation but at some point you’re chasing noise and picking winners, which leads to overestimates of magnitudes of effects.
So those are my thoughts. My goal here is not to “debunk” but to understand and quantify.
The paper (which at time was in preprint form) seemed potentially important, and I sent the following email to the first author:
Dear Dr. Gertler:
I read with interest your recent paper on the Jamaica experiment, and I had some thoughts; see here:
http://www.symposium-magazine.com/childhood-intervention-and-earnings/
And, for further background, see here:
http://andrewgelman.com/2013/11/05/how-much-do-we-trust-this-claim-that-early-childhood-stimulation-raised-earnings-by-42/
I suspect that, because of selection issues, your estimate of a 42% effect is an overestimate. Do you have any thoughts on this? In any case, I hope these comments will be helpful in your future work in this area.
Yours,
…
I received no response, which is fair enough: I’m sure the guy is busy and he certainly has no obligation to respond to emails. But, just in case, I passed him the message two more times, once through a friend who works in his department (and my friend confirmed that Gertler did receive the message) and once in response to a message that someone else had sent to Gertler, cc-ing me. I was frustrated to not hear anything back—after all, if someone says your estimate is too high, that’s a big deal, no?—but at least I was happy that the message got through.
Then more recently I came across this press release:
In the Friday (May 30) edition of the journal Science, researchers find that early childhood development programs are particularly important for disadvantaged children in Jamaica and can greatly impact an individual’s ability to earn more money as an adult. . . . Results from the Jamaica study show substantially greater effects on earnings than similar programs in wealthier countries. “We now have tangible proof of the potential benefits of early childhood stimulation and the importance of parenting in a developing country. . . .” Gertler said.
Annoyingly enough, the press release did not link to the published article—what’s with that, anyway???—but I did a google search and found it. From the abstract:
A substantial literature shows that U.S. early childhood interventions have important long-term economic benefits. However, there is little evidence on this question for developing countries. . . . the intervention increased earnings by 25%.
Cool! They listened to me! Or maybe it was just a coincidence. But, in any case, the estimated effect went down from 42% to 25%. We hear a lot about the “decline effect,” but in this case we’re seeing a decline from the pre-publication to the publication version of the same paper. That’s good news.
I’m not actually sure what Gertler et al. did to make the estimate fall from 42% to 25%. Part of it seems to be that they’re only looking at a subset of the data (“full-time jobs”) that happens to show a lower point estimate. But, compared to their earlier analysis, all the numbers seem to be lower, so they must have changed the analysis in some other way. I don’t see any discussion of the earlier 42% number in the article but maybe this comes up in the supplementary material.
Some other things
From the press release: “This study adds to the body of evidence, including Head Start and the Perry Preschool programs carried out from 1962-1967 in the U.S., demonstrating long-term economic gains from investments in early childhood development.” But, as I wrote on an earlier post on the topic, there is some skepticism about those earlier claims:
Here’s Charles Murray:
The most famous evidence on behalf of early childhood intervention comes from the programs that Heckman describes, Perry Preschool and the Abecedarian Project. The samples were small. Perry Preschool had just 58 children in the treatment group and 65 in the control group, while Abecedarian had 57 children in the treatment group and 54 in the control group. In both cases the people who ran the program were also deeply involved in collecting and coding the evaluation data, and they were passionate advocates of early childhood intervention.
Murray continues with a description of an attempted replication, a larger study of 1000 children that reported minimal success, and concludes:
To me, the experience of early childhood intervention programs follows the familiar, discouraging pattern …small-scale experimental efforts staffed by highly motivated people show effects. When they are subject to well-designed large-scale replications, those promising signs attenuate and often evaporate altogether.
Heckman replies here. I’m not convinced by his reply on this particular issue, but of course he’s an expert in education research so his article is worth reading in any case.
The other thing that’s buggin me is the following juxtaposition:
From the published article: “A substantial literature shows that U.S. early childhood interventions have important long-term economic benefits.”
From the press release: “Results from the Jamaica study show substantially greater effects on earnings than similar programs in wealthier countries. Gertler said this suggests that early childhood interventions can create a substantial impact on a child’s future economic success in poor countries.”
I don’t get it. On one hand they say they already knew that early childhood interventions have big effects in the U.S. On the other hand they say their new result shows “substantially greater effects on earnings.” I can believe that their point estimate of 25% is substantially higher than point estimates from other studies, or maybe that other studies showed big economic benefits but not big gains on earnings? In any case I can only assume that there’s a lot of uncertainty in this estimated difference.
Applying the Edlin factor?
My economist friend Aaron Edlin asked if I had a rule for routinely scaling down published estimates. I hemmed and hawed and wouldn’t give a general number. But I did write that I think an Edlin factor or 1/5 to 1/2 is probably my best guess for that Jamaica intervention example. In this case, Gertler et al. already went most of the way. The question is: had they originally reported 25% rather than 42%, what would I have said? I’m not sure.
The post Estimated effect of early childhood intervention downgraded from 42% to 25% appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
