Most data science projects are well served by a random test/train split. In our book Practical Data Science with R we strongly advise preparing data and including enough variables so that data is exchangeable, and scoring classifiers using a random test/train split. With enough data and a big enough arsenal of methods, it’s relatively easy … Continue reading Random Test/Train Split is not Always Enough → Related posts: Does Balancing…
As we’ve mentioned on previous occasions, one of the defining characteristics of data science is the emphasis on the availability of “large” data sets, which we define as “enough data that statistical efficiency is not a concern” (note that a “large” data set need not be “big data,” however you choose to define it). In … Continue reading Estimating Generalization Error with the PRESS statistic → Related posts: Don’t use…
Controlled experiments embody the best scientific design for establishing a causal relationship between changes and their influence on user-observable behavior. – Kohavi, Henne, Sommerfeld, “Practical Guide to Controlled Experiments on the Web” (2007) A/B tests are one of the simplest ways of running controlled experiments to evaluate the efficacy of a proposed improvement (a new […] Related posts: Unit tests as penance How to test XCOM “dice rolls” for fairness…
It occurred to us recently that we don’t have any articles about Bayesian approaches to statistics here. I’m not going to get into the “Bayesian versus Frequentist” war; in my opinion, which style of approach to use is less about philosophy, and more about figuring out the best way to answer a question. Once you […] Related posts: Statistics to English Translation, Part 2a: ’Significant’ Doesn’t Always Mean ’Important’ Worry…
In data science work you often run into cryptic sentences like the following: Age adjusted death rates per 10,000 person years across incremental thirds of muscular strength were 38.9, 25.9, and 26.6 for all causes; 12.1, 7.6, and 6.6 for cardiovascular disease; and 6.1, 4.9, and 4.2 for cancer (all P < 0.01 for linear […] Related posts: Level fit summaries can be tricky in R How to test XCOM…
XCOM: Enemy Unknown is a turn based video game where the player choses among actions (for example shooting an alien) that are labeled with a declared probability of success. Image copyright Firaxis Games A lot of gamers, after missing a 80% chance of success shot, start asking if the game’s pseudo random number generator is [...] Related posts: Statistics to English Translation, Part 2a: ’Significant’ Doesn’t Always Mean ’Important’ Statistics…
What is R2? In the context of predictive models (usually linear regression), where y is the true outcome, and f is the model’s prediction, the definition that I see most often is: In words, R2 is a measure of how much of the variance in y is explained by the model, f. Under “general conditions”, [...] Related posts: The Simpler Derivation of Logistic Regression Living in A Lognormal World “I…
Nina Zumel recently gave a very clear explanation of logistic regression ( The Simpler Derivation of Logistic Regression ). In particular she called out the central role of log-odds ratios and demonstrated how the “deviance” (that mysterious quantity reported by fitting packages) is both a term in “the pseudo-R^2″ (so directly measures goodness of fit) [...] Related posts: The Simpler Derivation of Logistic Regression Learn Logistic Regression (and beyond) Large…
Logistic regression is one of the most popular ways to fit models for categorical data, especially for binary response data. It is the most important (and probably most used) member of a class of models called generalized linear models. Unlike linear regression, logistic regression can directly predict probabilities (values that are restricted to the (0,1) [...] Related posts: The equivalence of logistic regression and maximum entropy models Learn Logistic Regression…
