Category: Decision Theory

When LOO and other cross-validation approaches are valid

Introduction Zacco asked in Stan discourse whether leave-one-out (LOO) cross-validation is valid for phylogenetic models. He also referred to Dan’s excellent blog post which mentioned iid assumption. Instead of iid it would be better to talk about exchangeability assumption, but I (Aki) got a bit lost in my discourse answer (so don’t bother to go […]

The post When LOO and other cross-validation approaches are valid appeared first on Statistical Modeling, Causal Inference, and Social Science.

When LOO and other cross-validation approaches are valid

Introduction Zacco asked in Stan discourse whether LOO is valid for phylogenetic models. He also referred to Dan’s excellent blog post which mentioned iid assumption. Instead of iid it would be better to talk about exchangeability assumption, but I (Aki) got a bit lost in my discourse answer (so don’t bother to go read it). […]

The post When LOO and other cross-validation approaches are valid appeared first on Statistical Modeling, Causal Inference, and Social Science.

“Seeding trials”: medical marketing disguised as science

Paul Alper points to this horrifying news article by Mary Chris Jaklevic, “how a medical device ‘seeding trial’ disguised marketing as science.” I’d never heard of “seeding trials” before. Here’s Jaklevic: As a new line of hip implants was about to be launched in 2000, a stunning email went out from the manufacturer’s marketing department. […]

The post “Seeding trials”: medical marketing disguised as science appeared first on Statistical Modeling, Causal Inference, and Social Science.

“Seeding trials”: medical marketing disguised as science

Paul Alper points to this horrifying news article by Mary Chris Jaklevic, “how a medical device ‘seeding trial’ disguised marketing as science.” I’d never heard of “seeding trials” before. Here’s Jaklevic: As a new line of hip implants was about to be launched in 2000, a stunning email went out from the manufacturer’s marketing department. […]

The post “Seeding trials”: medical marketing disguised as science appeared first on Statistical Modeling, Causal Inference, and Social Science.

How dumb do you have to be…

I (Phil) just read an article about Apple. Here’s the last sentence: “Apple has beaten earnings expectations in every quarter but one since March 2013.”
[Note added a week later: on July 31 Apple reported earnings for the fiscal third…

Parsimonious principle vs integration over all uncertainties

tl;dr If you have bad models, bad priors or bad inference choose the simplest possible model. If you have good models, good priors, good inference, use the most elaborate model for predictions. To make interpretation easier you may use a smaller model with similar predictive performance as the most elaborate model. Merijn Mestdagh emailed me […]

The post Parsimonious principle vs integration over all uncertainties appeared first on Statistical Modeling, Causal Inference, and Social Science.

Parsimonious principle vs integration over all uncertainties

tl;dr If you have bad models, bad priors or bad inference choose the simplest possible model. If you have good models, good priors, good inference, use the most elaborate model for predictions. To make interpretation easier you may use a smaller model with similar predictive performance as the most elaborate model. Merijn Mestdagh emailed me […]

The post Parsimonious principle vs integration over all uncertainties appeared first on Statistical Modeling, Causal Inference, and Social Science.

On deck through the rest of the year

July: The Ponzi threshold and the Armstrong principle Flaws in stupid horrible algorithm revealed because it made numerical predictions PNAS forgets basic principles of game theory, thus dooming thousands of Bothans to the fate of Alderaan Tutorial: The practical application of complicated statistical methods to fill up the scientific literature with confusing and irrelevant analyses […]

The post On deck through the rest of the year appeared first on Statistical Modeling, Causal Inference, and Social Science.

The “Psychological Science Accelerator”: it’s probably a good idea but I’m still skeptical

Asher Meir points us to this post by Christie Aschwanden entitled, “Can Teamwork Solve One Of Psychology’s Biggest Problems?”, which begins: Psychologist Christopher Chartier admits to a case of “physics envy.” That field boasts numerous projects on which international research teams come together to tackle big questions. Just think of CERN’s Large Hadron Collider or […]

The post The “Psychological Science Accelerator”: it’s probably a good idea but I’m still skeptical appeared first on Statistical Modeling, Causal Inference, and Social Science.

Ways of knowing in computer science and statistics

Brad Groff writes: Thought you might find this post by Ferenc Huszar interesting. Commentary on how we create knowledge in machine learning research and how we resolve benchmark results with (belated) theory. Key passage: You can think of “making a a deep learning method work on a dataset” as a statistical test. I would argue […]

The post Ways of knowing in computer science and statistics appeared first on Statistical Modeling, Causal Inference, and Social Science.

Answering the question, What predictors are more important?, going beyond p-value thresholding and ranking

Daniel Kapitan writes: We are in the process of writing a paper on the outcome of cataract surgery. A (very rough!) draft can be found here, to provide you with some context:  https://www.overleaf.com/read/wvnwzjmrffmw. Using standard classification methods (Python sklearn, with synthetic oversampling to address the class imbalance), we are able to predict a poor outcome […]

The post Answering the question, What predictors are more important?, going beyond p-value thresholding and ranking appeared first on Statistical Modeling, Causal Inference, and Social Science.

Chasing the noise in industrial A/B testing: what to do when all the low-hanging fruit have been picked?

Commenting on this post on the “80% power” lie, Roger Bohn writes: The low power problem bugged me so much in the semiconductor industry that I wrote 2 papers about around 1995. Variability estimates come naturally from routine manufacturing statistics, which in semicon were tracked carefully because they are economically important. The sample size is […]

The post Chasing the noise in industrial A/B testing: what to do when all the low-hanging fruit have been picked? appeared first on Statistical Modeling, Causal Inference, and Social Science.

About that quasi-retracted study on the Mediterranean diet . . .

Some people asked me what I thought about this story. A reporter wrote to me about it last week, asking if it looked like fraud. Here’s my reply: Based on the description, there does not seem to be the implication of fraud. The editor’s report mentioned “protocol deviations, including the enrollment of participants who were […]

The post About that quasi-retracted study on the Mediterranean diet . . . appeared first on Statistical Modeling, Causal Inference, and Social Science.