Posts Tagged ‘ Significance ’

The reality is most A/B tests fail, and Facebook is here to help

April 16, 2014
By

Two years ago, Wired breathlessly extolled the virtues of A/B testing (link). A lot of Web companies are in the forefront of running hundreds or thousands of tests daily. The reality is that most A/B tests fail. A/B tests fail for many reasons. Typically, business leaders consider a test to have failed when the analysis fails to support their hypothesis. "We ran all these tests varying the color of the…

Read more »

Round-up of coverage of the Big Miss of Big Data

April 9, 2014
By

There is now some serious soul-searching in the mainstream media about their (previously) breath-taking coverage of the Big Data revolution. I am collecting some useful links here for those interested in learning more. Here's my Harvard Business Review article in which I discussed the Science paper disclosing that Google Flu Trends, that key exhibit of the Big Data lobby, has systematically over-estimated flu activity for 100 out of the last…

Read more »

Drowning in insignificance

February 26, 2014
By
Drowning in insignificance

Some researchers (in both science and marketing) abuse a slavish view of p-values to try and falsely claim credibility. The incantation is: “we achieved p = x (with x ≤ 0.05) so you should trust our work.” This might be true if the published result had been performed as a single project (and not as […] Related posts: Bayesian and Frequentist Approaches: Ask the Right Question Worry about correctness and…

Read more »

An example of why data will not end debate

February 14, 2014
By
An example of why data will not end debate

One oft-repeated "self-evident" tenet of Big Data is that data end all debate. Except if you have ever worked for a real company (excluding those ruled by autocrats), and put data on the table, you know that the data do not end anything. Reader Ben M. sent me to this blog post by Benedict Evans, showing a confusing chart showing how Apple has "passed" Microsoft. Evans used to be a…

Read more »

MailChimp Gmail study as an example of Big Data studies 2/2

February 4, 2014
By
MailChimp Gmail study as an example of Big Data studies 2/2

Previously, I analyzed the data analysis by MailChimp on the impact of Gmail's new tabbing feature, and noted a potential data issue (link). In Part 2, I will look at the MailChimp study as a typical example of "Big Data" studies. The Gmail study has several features that are hallmarks of Big Data. First and foremost, the analyst boasts of a staggering amount of data ("29 billion emails, 4.9 billion…

Read more »

Introducing the statistical parable

January 27, 2014
By

Last week, Andrew Gelman (link) and I were kindred spirits: we both did a "numbersensing" exercise on two different data analyses. I was reading the MailChimp study on the effect of Google siphoning off "marketing" emails into a separate tab, and a noise buzzed my head when I saw that the aggregate click-to-open ratio was reported at an inconceivable 85%. (See Part 1 of my reaction here.) In the meantime,…

Read more »

Avinash’s magnificent zoo, and the unfulfilled promise of Big Data

January 8, 2014
By

Avinash (Web Analytics 2.0) is fond of animal metaphors. I think he's the one who coined HIPPOs (Highest Paid Person's Opinion). Now he's come up with Reporting Squirrels and Analysis Ninjas. See his recent post here. In short, he is shouting about "return on analytics", a really, really important thing. What is being lost in the hype of Big Data is that all the investment in analytics has to generate…

Read more »

Story time on aspirin

December 4, 2013
By

USA Today, who brought us a dubious early signaling system for sudden cardiac arrest in a prior post, also gave us “Story Time!” on aspirin on the same day (link). The article started with the conclusion: “Taking an aspirin before bed may reduce the chance of having a heart attack or stroke in the morning”. What is the data used to support this cause—effect statement? *** The researchers looked at…

Read more »

Un peu plus près des étoiles (***)

November 15, 2013
By
Un peu plus près des étoiles (***)

Il y a eu un gros buzz, récement autour du papier de Valen Johnson paru dans PNAS. L’article a été repris un peu partout (http://nature.com/news/, http://blogs.scientificamerican.com/absolutely-maybe/, http://arstechnica.com/science/ ou encore http://passeurdesciences.blog.lemonde.fr/ qui a repris l’information, en français). Et plusieurs personnes m’ont fait suivre des liens, en me demandant mon avis, par courriel ou via twitter. Je ne vais pas revenir sur l’étude (pour l’instant) ni sur les mauvaises lectures de l’étude, mais plutôt sur le buzz…

Read more »

Numbersense Pros: Interview with Andrew Gelman

September 12, 2013
By
Numbersense Pros: Interview with Andrew Gelman

Professor Andrew Gelman is a pioneer in statistics blogging. His blog is one of my regular reads, a mixture of theoretical pieces, applied work, psychological musing, rants about unethical academics, advocacy of statistical graphics, and commentary on literature. He's one of the few statisticians who gets opinion pieces published in the New York Times. His expertise is statistics in politics, but I also enjoy his work on the stop-and-frisk policies…

Read more »


Subscribe

Email:

  Subscribe