One statistical concept that instructors frequently don't have time to cover in Stat 101 is the "interaction" effect. I will explain this concept using the fantastic interactive graphic by the visualization team at the German publication Zeit (please also read the corresponding post on Junk Charts here for some background.) When we ignore interactions, we end up with overly simplistic statistical summaries. For example, some study might find that drinking…
A lot of Big Data analyses default to analyzing count data, e.g. number of searches of certain keywords, number of page views, number of clicks, number of complaints, etc. Doing so throws away much useful information, and frequently leads to bad analyses. *** I was reminded of the limitation of count data when writing about the following chart, which I praised on my sister blog as a good example of…
This is a cross-post to both my blogs. Thanks to the ~200 or so people who showed up at last week's Data Scientist Meetup in Cambridge, Mass., hosted by John Baker. I gave a brief introduction to the concept of "numbersense", and was part of a panel of "chief data scientists" talking about how to run data teams. Thanks to those who asked questions. This month, I am back in…
In case you are not subscribed to my dataviz feed, I put up a post yesterday that is highly relevant to readers here interested in statistical topics. The post discusses a graphic of a New York Times article that interprets the official inflation rate (known as the CPI). I devoted an entire chapter of Numbersense (link)to the question of why the official inflation rate diverges from our everyday experience. In…
For those who weren't able to attend my recent talks, a few have surfaced online. *** JMP put up the video of the webcast from last Friday with Alberto Cairo, a data visualization expert and author of The Functional Art. You can access it from here. This event is part of their Analytically Speaking series with recent guests such as David Hand and Michael Schrage. I also appear on this…
Long before I came up with "numbersense," I wrote about "true lies" in data analysis. (link) The nature of data, especially Big (as in multidimensional) Data, is that one can come up with an infinite number of statistical computations, all of which are "true" in the sense that one would obtain such statistics were one to plug the data into textbook formulas. Inevitably, some of these statistics lead to contradictions.…
One oft-repeated "self-evident" tenet of Big Data is that data end all debate. Except if you have ever worked for a real company (excluding those ruled by autocrats), and put data on the table, you know that the data do not end anything. Reader Ben M. sent me to this blog post by Benedict Evans, showing a confusing chart showing how Apple has "passed" Microsoft. Evans used to be a…
First, I saw Andrew Gelman's rant about "big bad education" (link) which leads me to Mark Palko's rant about teaching "the Law of Large Numbers" in the new "Common Core" curriculum for New York schools. Mark's conclusion being: If we start talking about setting aside significant time to cover probability and statistics accurately and in reasonable depth and put the ideas in proper context, you have my enthusiastic support, but…
Some interviews with me or snippets of such have surfaced recently. Here is a list: Kate Meersschaert interviewed me for New Learning Times (link; registration required). I talked about my teaching philosophy, and why I write books. Jay Ulfelder, a political scientist who keeps an interesting blog, recommends Numbers Rule Your World, and a few other books for political scientists (link). If you haven't heard already, 2013 is the International…
Another statistician has joined the blogosphere, and I highly recommend his blog. The name is Good Stats Bad Stats, and the author is Larry Cahoon who is retired after a career in government (official) statistics. Larry is a long-time reader of Junk Charts as well, and I'm late to the party, as he has been blogging for almost a year. Here are some pieces that caught my attention: In this…
