(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Gur Huberman points us to this post by Joshua Gans, “A few words on Twitter’s 280 experiment.” I hate twitter but I took a look anyway, and I’m glad I did, as Gans makes some good points and some bad points, and it’s all interesting.
Gans starts with some intriguing background:
Twitter have decided to run an experiment. They are giving random users twice the character limit — 280 rather than 140 characters. Their motivation was their observation that in Japanese, Korean and Chinese 140 characters conveys alot more information and so people tend to tweet more often. Here is their full statement.
The instructive graph is this:
The conclusion drawn is that Japanese tweeters do not hit their character limit as much as English tweeters. They also claim they see more people tweeting in the less constrained languages. Their conclusion is that not having as tight a character limit makes expression easier and so you get more of it.
Interesting. Gans continues:
What Twitter have just told us is that the world gave them a natural experiment and they liked what they saw. . . . What was Twitter’s reaction to this? To do an experiment. In other words, they are worried that the natural experiment isn’t telling them enough. Since it is about as clean a natural experiment as you are likely to get in society, we can only speculate what they are missing. Are they concerned that this is something cultural? (They had three cultures do this so that is strange). Moreover, many of those users must also speak English so one has to imagine something could be learned from that.
I’m not quite sure what he means by a “culture,” but this generally seems like a useful direction to explore. One thing, though: Gans seems to think it’s a big mystery why Twitter would want to do an experiment rather than just draw inferences from observational data. But an experiment here is much different from the relevant observational data. In the observational data, the U.S. condition is unchanged; in the experiment, the U.S. condition is changed. That’s a big deal! We’re talking about two different comparisons:
observational: U.S. with a 140 character limit vs. Japan with a 140 character limit.
experimential: U.S. with a 140 character limit vs. U.S. with a 280 character limit.
These comparisons are a lot different! It doesn’t matter how “clean” is the observational comparison (which I think Gans somewhat misleadingly calls a “natural experiment”); these are two different comparisons.
Gans continues:
My point is: the new experiment must be testing a hypothesis. But what is that hypothesis?
Huh? There’s no requirement at all that an experiment “must be testing a hypothesis.” An experiment is a way to gather data. You can use experimental data to test hypotheses, or to estimate parameters, or to make predictions, or to make decisions. All these can be useful. But none of them is necessary. In particular, I’m guessing that Twitter wants to make decisions (also to get some publicity, goodwill, etc.). No need for there to be any testing of a hypothesis.
Gans does have some interesting thoughts on the specifics:
The obvious way [to do an experiment] would be to announce, say, a three month trial across the whole of English speaking twitter and observe changes. That would replicate the natural experiment to a degree. Or, alternatively, you might pick a language with a small number of users and conduct the experiment there. . . .
That is not what Twitter did. They decided to randomise across a subset of English users — giving them 280 characters — and leaving the rest out. That strikes me as a bad idea because those random people are not contained. They mix with the 140 people. . . .
Why is this a terrible idea? Because it is not an experiment that tests what Twitter was likely missing from the information they gained already. Instead, it is an experiment that tests the hypothesis — what if we gave some people twice the limit and threw all of them together with those without? The likelihood that Twitter learns anything with confidence to move to a 280 limit from everyone is very low from this.
All this seems odd to me. Gans’s concern is spillover, and that’s a real concern, but any design has issues. His proposed three-month trial has no spillover but is confounded with time trends. If it’s not one thing it’s another. My point is that I don’t think it’s right to say that a design is “terrible” just because there’s spillover, any more than you should say that the design is terrible if it is confounded with time, any more than you should describe an observational comparison which is confounded with country as if it is “as clean as you are likely to get.”
Yes, identify the problems in data and consider what assumptions are necessary to learn from these problems. No, don’t be so sure that what people are doing is a bad idea. Remember that Twitter has goals beyond testing hypotheses—indeed I’d guess that Twitter isn’t interested in hypothesis testing at all! It’s a business decision and Twitter has lots of business goals. Just to start, see this comment from Abhishek on the post in question.
Finally, Gans writes:
What we should be complaining about is why they are running such an awful experiment and how they came to such a ludicrous decision on that.
Huh? We should be complaining because a company is suboptimally allocating resources? I don’t get it. We can laugh at them, but why complain?
P.S. Yes, I recognize the meta-argument, that if I think Gans has no reason to complain that Twitter did an experiment that’s different from the experiment he would’ve preferred, then, similarly, I have no reason to complain that Gans wrote a blog post different from the post that I would’ve preferred. Fair enough.
What I’m really saying here is that I disagree with much of what Gans writes. Or, to be more precise, I like Gans’s big picture—he’s looking at a data analysis (the above graph) and thinking of it as an observational study, and he’s looking at a policy change (the 280-character rule) and thinking of it as an experiment—but I think he’s getting stuck in the weeds, not fully recognizing the complexity of the situation and thinking that there’s some near-ideal experiment and hypothesis out there.
I appreciate that Gans is stepping back, taking a real-world business decision that’s in the news and trying to evaluate from first principles. We certainly shouldn’t assume that any decision made by Twitter, say, is automatically a wise choice, nor should we assume that change is bad. It’s a good idea to look at a policy change and consider what can be learned from it. (For more on this point, see Section 4 of this review.) I’d just like to step back a few paces further and place this data gathering in the context of various goals of Twitter and its users.
So I thank Gans for getting this discussion started, and I thank Huberman for passing it over to us.
P.P.S. I wrote this post in Sep 2017 and it’s scheduled to appear in Apr 2018, at which time, who knows, tweets might be 1000 characters long. I still prefer blogs.
The post A few words on a few words on Twitter’s 280 experiment. appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
