StatsBlogs

What is the (expensive) miracle drug that cut colon cancer deaths by 20%?

junkcharts — Fri, 11 May 2012 03:38:46 +0000

(This article by junkcharts was originally published at Numbers Rule Your World.)

Andrew Gelman linked to this great reporting by Reuters on U.S. healthcare economics. It's a must-read. Be patient, and read through to the end even though it's a long piece.

Andrew cites statistician Don Berry who explains what "lead time bias" is, and why survival time is always the wrong metric to use in evaluating health outcomes. Survival time is the time from diagnosis to death. By doing more screening and diagnosing earlier, survival time will magically increase even if the patient's life expectancy stays put.

***

I ignored Andrew's warning and spent some time reading the Philipson, et. al. paper (link). Time that I want back but couldn't. To save you the trouble, I will discuss a few gaping holes other than the howler already identified by Berry - there are many other less significant issues.

The title of the paper purports to address the following "causal" relationship:

The reader immediately discovers that the authors analyzed a different "causal" relationship:

It may appear that the substitutions are harmless: spending on cancer care is a proxy for overall healthcare spending; survival gains for cancer patients is a proxy for overall health benefits. The authors hid the useful information in the Appendix (available online). In Table 3, we learn that spending on cancer care is only a single-digit percentage of total health care spending in almost every country. Besides, the total deaths by the 13 types of cancers counted in their study constitute only 31 percent of the total cancer deaths in the U.S. (using the 2011 statistics from this report - PDF). This list of included cancer types excludes the biggest killer (lung, over 150,000 deaths) while it includes testis which caused 350 deaths in 2011.

So, even if the analysis is correct, the result cannot be generalized to talk about cost and benefit of all health care spending. This is an instance of "availability bias": even though cancer makes a lot of news, most health care spending has nothing to do with cancer, and so we can't use cancer care spending as a proxy.

***

In assessing the value of cancer care spending, the authors decided to use a modeled change in death rates, rather than the actual observed data. Neither in the paper nor in the appendix is the actual model reported, nor is there any information on goodness of fit. However, we don't need to know the model to know it doesn't fit.

Take a look at the fourth column of Table 1 in the Appendix. This column shows the predicted deaths avoided or incurred in the U.S. (given the additional spending in the U.S. relative to "Europe").

Let's do a sanity check on these numbers. For colorectal cancer, the model claims that the extra spending has avoided 282,000 deaths over the 23 years (1982-2005), or roughly 12,300 deaths per year. According to the cancer death statistics, about 50,000 deaths from colon cancer actually occurred in the U.S. in 2011. That means the model claims that colon cancer deaths would have been 25% higher were it not for the extra spending. What is the miracle drug that caused this gigantic improvement? What prevents this amazing new treatment from crossing the Atlantic?

Maybe you believe in miracles. Then, take a look at stomach cancer. Here, the negative number seems to imply that the additional spending has induced 225,000 stomach cancer deaths over 23 years. That sounds really horrifying. Given that stomach cancer killed 10,300 Americans in 2011, the model claims that the extra spending has doubled the number of deaths from stomach cancer!

Simply put, their model makes no sense.

***

Now, go back to Table 3 in the Appendix and read the note. It says that missing data for percentage of health care spending that is cancer related is imputed as 6.5% (30% higher than the U.S. assumption of 5% which came from a totally different source), and we find that Iceland, Norway, Slovkia and Slovenia (40% of the countries) are all imputed.

The problem here is that the authors are not consistent in their treatment of missing data. In the main paper, they explain again and again that their sample of data is restricted by data availability (i.e. they didn't impute values for missing data). For example, the choice of the 10 European countries is because "only ten reported data consistently over the 1983-99 period". This means no Italy, no Spain but you have Wales and Scotland (but no England), and also Slovakia and Slovenia (why are they comparable to U.S.?).

Why those particular 13 cancers? Because "data were consistently available from both the European and US survival databases". This means including testis cancer and excluding lung cancer. Insteading of imputing values for lung cancer, they just drop the cancer type that causes the most deaths.

Why look at survival differences only for patients diagnosed from 1995 through 1999? You guessed it. It's because only in those periods can they find consistent data.

Given that they use models throughout the research, and they imputed values for proportion of spending on cancer treatment, they could have tried to impute values in these other decisions, and then the result could perhaps be generalized.

Dropping data because some variables are missing should be justified clearly. It's too easy to cherry-pick your dataset this way.

***

How about another non-sensical assumption? The average value of an additional year of life of someone who's dying from cancer is $150,000 to $360,000. They describe this as "standard figures for an extra year of life" and call the lower end of the range "conservative". Only 5% of Americans earn over $100,000 per year. The median personal income is less than $40,000. (From Wikipedia, for 2004, I think). Enough said.

***

It's sad that this paper gets publicity only because it makes a conclusion that is against "conventional wisdom". The clear evidence so far has been that while the U.S. spends twice as much on health care as other "wealthy" nations, our life expectancy is lower, and the bottom of the class. (See here, for example.)

The chart shown on the right is as clear as it can be. (I discussed this chart on Junk Charts.) The situation with science journalism is very dire, in my opinion, when outlets are chasing clicks and sales by publicizing bad studies that have eye-catching headlines.

Please comment on the article here: Numbers Rule Your World

What are the products of data analysis?

rdpeng — Thu, 10 May 2012 12:55:41 +0000

(This article by rdpeng was originally published at Simply Statistics.)

Thanks to everyone for the feedback on my post on knowing when someone is good at data analysis. A couple people suggested I take a look here for a few people who have proven they’re good at data analysis. I think that’s a great idea and a good place to start.

But I also think that while demonstrating an ability to build good prediction models is impressive and definitely shows an understanding of the data, not all important problems can be easily posed as prediction problems. Most of my work does not involve prediction at all and the problems I face (i.e., estimating very small effects in the presence of large unmeasured confounding factors) would be difficult to formulate as a prediction challenge (at least, I can’t think of an easy way). In fact, part of my and my colleagues’ research involves showing how statistical methods designed for prediction problems can fail miserably when applied to other non-prediction settings.

The general question I have is what is a useful product that you can produce from a data analysis that demonstrates the quality of that analysis? So, a very small mean squared error from a prediction model would be one product (especially if it were smaller than everyone else’s). Maybe a cool graph with a story behind it?

If I were hiring a musician for an orchestra, I wouldn’t have to meet that person to have strong evidence that he/she were good. I could just listen to some recordings of that person playing and that would be a pretty good predictor of how that person would perform in the orchestra. In fact, some major orchestras do completely blind auditions so that although the person is present in the room, all you hear is the sound of the playing.

What seems to be true with music at least, is that even though the final performance doesn’t specifically reveal the important decisions that were made along the way to craft the interpretation of the music, somehow one is still able to appreciate the fact that all those decisions were made and they benefitted the performance. To me, it seems unlikely to arrive at a sublime performance either by chance or by some route that didn’t involve talent and hard work. Maybe it could happen once, but to produce a great performance over and over requires more than just luck.

What products could you send to someone to convince them you were good at data analysis? I raise this question primarily because when I look around at the products that I make (research papers, software, books, blogs), even if they are very good, I don’t think they necessarily convey any useful information about my ability to analyze data.

What’s the data analysis equivalent of a musician’s performance?

Please comment on the article here: Simply Statistics

DealBook: Glaxo to Make Hostile Bid for Human Genome Sciences

rdpeng — Wed, 09 May 2012 13:48:59 +0000

(This article by rdpeng was originally published at Simply Statistics.)

DealBook: Glaxo to Make Hostile Bid for Human Genome Sciences:

On the heals of the failed Illumina deal, another bare-knuckled brawl.

GlaxoSmithKline plans to take its bid for Human Genome Sciences directly to shareholders this week, after being rejected by the biotechnology company last month.

Please comment on the article here: Simply Statistics

Book “R and Data Mining: Examples and Case Studies” on CRAN

Yanchang Zhao — Wed, 09 May 2012 09:01:48 +0000

(This article by Yanchang Zhao was originally published at RDataMining.)

by Yanchang Zhao, RDataMining.com

My book in draft titled “R and Data Mining: Examples and Case Studies” is now available on CRAN at http://cran.r-project.org/other-docs.html. It is scheduled to be published by Elsevier in late 2012. Its latest version can be downloaded at http://www.rdatamining.com/docs.

The book presents many examples on data mining with R, including data exploration, decision trees, random forest, regression, clustering, time series analysis & mining, and text mining. Some other chapters in progress are social network analysis, outlier detection, association rules and sequential patterns.

If you find any errors in the book, or have any comments or suggestions, please feel free to let me know at . Feedback from you will help to make it better. Thanks.

Please comment on the article here: RDataMining

How to report a Bayesian analysis: Teaching why

John K. Kruschke — Wed, 09 May 2012 02:55:00 +0000

(This article by John K. Kruschke was originally published at Doing Bayesian Data Analysis.)

An interesting recent article emphasized that asking students to generate what should be reported from a Bayesian analysis also gets them to think about why things need to be reported, which in turn can get students to understand better how Bayesian analysis works and what it means. In this blog post I briefly summarize the article, compare it with the recommendations for reporting a Bayesian analysis in Doing Bayesian Data Analysis, and suggest that there are some important points for reporting that might be difficult for beginning students to generate on their own.

The article is: Pullenayegum, E. M., Guo, Q., & Hopkins, R. B. (2012). Developing critical thinking about reporting of Bayesian analyses. Journal of Statistics Education, 20(1). It's available at this link. Here are some excerpts:

[Abstract:] Whilst there are published guidelines on reporting of Bayesian analyses, students should also be encouraged to think about why some items need to be reported whereas others do not. We describe a classroom activity in which students develop their own reporting guideline. ... [Section 1:] Since Bayesian analyses are far less widely used in medicine [than frequentist analyses], it is difficult for students to gauge what is typical. Thus, teaching students how to report Bayesian analyses in the medical literature is an important component of a course in Bayesian biostatistics. When teaching reporting, we need to avoid training students to follow a set of rules unthinkingly. Rather we need to focus on helping students to think critically about what is needed. That is, we need to teach not just the “what”, but also the “why”. ... [Section 4.4:] Two of us (RBH and QG) participated in the first implementation of the exercise as students. As the exercise progressed, we found ourselves thinking as readers, reviewers or editors rather than as the statistician or author. ... [Section 5:] The value of the activity is in the learning process rather than the guideline itself.

The exercise itself consists of students anonymously generating candidate items for inclusion, then having the items collated onto a master list of candidates, then rating the items for importance and discussing why they are important. (See the article for details about the group dynamic.) Importantly, but perhaps not emphasized enough in the article, the instructor can (and did) anonymously contribute items to the list during the first round. This strikes me as a very useful exercise for students, and I may try some variation of it myself next time I teach my course.

The article reports the lists of items generated by two different classes. The lists included points about the prior distributions, the likelihood function, the analysis technique, and the results. Please see Table 1 of the article for details. Although the emphasis of the article was on getting students to think about why items should or need not be reported, rather than on the actual list created, it is still informative to see what items the students settled on, and whether anything went missing that might be considered to be important.

In particular, it's natural for me to compare the student-generated lists with my own recommendations in Doing Bayesian Data Analysis (Ch. 23, pp. 620-622):

There is notable overlap with items in the student-generated lists, but the first two "essential" points raised in DBDA were not emphasized by the students. These points are (1) Motivate the use of Bayesian (non-NHST) analysis, and (2) Clearly describe the model and its parameters. Both of these points are about reporting the forest before reporting the trees. Both of these points are about contextualizing the analysis and giving it meaning. In particular, the second point entails the idea that the model is a particular description of data, selected from the space of all possible models because it is meaningful in the particular context. Finally, the "essential" points in DBDA are meant to be reported in sequence. This temporal, sequential order of points is important for the comprehensibility of the report. For example, the likelihood function and its parameters must be explained before the prior and its parameters can be explained. (And the structure of the data must be explained before the likelihood function can be described.)

I think that all these issues ---motivating use of Bayesian analysis, describing the model and parameters, and reporting in a particular sequential order--- are about understanding the larger context in which statistical analysis resides, and this larger context might be relatively difficult for beginning students to apprehend. Therefore instructors who adopt the exercise, of students generating guidelines for reporting, may want to seed the candidate list with these sorts of items, or at least be sure that they are brought up during discussion.

Please comment on the article here: Doing Bayesian Data Analysis

A simple example of parallel computing on a Windows (and also Mac) machine

Yanchang Zhao — Tue, 08 May 2012 09:55:12 +0000

(This article by Yanchang Zhao was originally published at RDataMining.)

by Yanchang Zhao, RDataMining.com

With a Mac, parallel computing can be achieved with package multicore. Unfortunately, it does not work under Windows.

A simple way for parallel computing under Windows (and also Mac) is using package snowfall, which can work with multi-CPU or multi-core on a single machine, as well as a cluster of multiple machines. For parallel computing on a single machine, it is simple and easy as below.

> library (snowfall)
# initialize cluster
> sfInit (parallel=TRUE , cpus=4)
# parallel computing
> result <- sfLapply(1:10, log)
# stop cluster
> sfStop ()

Simply replace “1:10″ and “log” with your parameter and function to make you own parallel computing. Function sfLapply() is a parallelized version of lapply(). Some other fuctions are sfSapply, sfApply, sfRapply and sfCapply.

Please comment on the article here: RDataMining

Data analysis competition combines statistics with speed

rdpeng — Tue, 08 May 2012 00:12:20 +0000

(This article by rdpeng was originally published at Simply Statistics.)

Data analysis competition combines statistics with speed:

Just a bit more alumni spirit on my end. Looks the data they were looking at was from kiva.org.

Please comment on the article here: Simply Statistics

How do you know if someone is great at data analysis?

rdpeng — Mon, 07 May 2012 13:17:16 +0000

(This article by rdpeng was originally published at Simply Statistics.)

Consider this exercise. Come up with a list of the top 5 people that you think are really good at data analysis.

There’s one catch: They have to be people that you’ve never met nor have had any sort of personal interaction with (e.g. email, chat, etc.). So basically people who have written papers/books you’ve read or have given talks you’ve seen or that you know through other publicly available information. Who comes to mind? It’s okay to include people who are no longer living.

The other day I was thinking about the people who I think are really good at data analysis and it occurred to me that they were all people I knew. So I started thinking about people that I don’t know (and there are many) but are equally good at data analysis. This turned out to be much harder than I thought. And I’m sure it’s not because they don’t exist, it’s just because I think good data analysis chops are hard to evaluate from afar using the standard methods by which we evaluate people.

I think there are a few reasons. First, people who are great at data analysis are likely not publishing papers or being productive in a manner that I, an outsider, would be able to observe. If they’re working at a pharmaceutical company working on a new drug or at some fancy new startup company, there’s no way I’m ever going to know about it unless I’m directly involved.

Another reason is that even for people who are well-known scientists or statisticians, the products they produce don’t really highlight the difficulties overcome in data analysis. For example, many good papers in the statistics literature will describe a new method with brief reference to the data that inspired the method’s development. In those cases, the data analysis usually appears obvious, as most things do after they’ve been done. Furthermore, papers usually exclude all the painful details about merging, cleaning, and inspecting the data as well as all the other things you tried that didn’t work. Papers in the substantive literature have a similar problem, which is that they focus on a scientific problem of interest and the analysis of the data is secondary.

As skills in data analysis become more important, it seems odd to me that we don’t have a great way to evaluate a person’s ability to do it as we do in other areas.

Please comment on the article here: Simply Statistics

Uncertainty, luck and control

Dr Nic — Mon, 07 May 2012 04:07:35 +0000

(This article by Dr Nic was originally published at Learn and Teach Statistics and Operations Research.)

Is probability mathematics or statistics?

Probability is often taught alongside statistics, and it is a rather uneasy alliance. Mathematics teachers like probability. It behaves as a good mathematical topic should, and gives nice exact answers. But it also provides the theoretical underpinnings to the rather murkier subjects of statistics, modeling and operations research. An understanding of probability is needed for simulation, queueing theory, inference, decision analysis, project management and inventory control.

Mathematics teachers should love probability as it is a subject that lends itself to hands-on activities and real-life applications. Problem is, the whole uncertain nature of probability means that you can’t guarantee that an experiment will give the results you want.

Sometimes chance activities don’t turn out as we want them to.

For example, one activity I use to teach about sampling variation is to get each member of the class to write down how many hours that week they have spent on the subject. I collect up the pieces of paper, and in front of them take two samples, each of size 5. I ask the class if they expect the means for the two samples to be the same. And they don’t, and usually they aren’t. But one time they were! Now how likely was that? That would be an interesting advanced exercise. I was able to turn it into a learning moment, but I would rather that the means had been different, to illustrate my point.

Probability is about what might happen in the future, and statistics is about what has already happened. Probability is predicting what you might draw out, from knowing what is in the bag, whereas statistics is inferring what is in the bag, from what you drew out. (I would like to attribute this great analogy but can’t remember where I found it – so if anyone can tell me, please do.)

When I teach about probability I introduce four sources of probability estimates: A priori, relative frequency, modeled and subjective. Often students are exposed only to a priori, and get stumped in areas like decision analysis or simulation where other types of probability are needed.

A priori

A priori describes the probability of coins, dice, balls and urns, of people drawing socks out of drawers in the dark and selecting chocolates with their eyes closed. A priori seems rather fantastic, when taken at that level. This is the kind of probability that maths teachers like. The probability of an event happening is the number of ways that event can happen divided by the total number of possible outcomes. You need addition and division, and the ability to turn expressions into words. Casino gambling and lotteries are based on a priori probabilities. There is exactly one chance in 33 that the ball will land on the number 7 in roulette. It is rather nifty really.

Relative frequency

Probability based on relative frequency draws on the past. In the past 20% of people have been late to appointments, so we use 0.2 or 20% as the estimate for how often we expect people to be late in the future. This estimation of probability is only as good as the data collected previously, and the stability of the system.

Modelled

We use models to try to predict outcomes

Weather, avalanches and earthquake events are assigned probabilities using historical data and mathematical models. Forecasting is a major part of business analysis, to try to reduce the uncertainty of the future by modeling with the data from the past. Relative frequency is a very simple form of modeled probability, where the parameter from the past is used unchanged.

Subjective

When we estimate the probability of an occurrence we produce a subjective probability figure. There is a whole branch of research into how people predict, and their systematic biases. Things like the vividness of the outcome, the number of similar occurrences you can think of, and a number given you ahead of time, all affect subjective probabilities. Even language can mislead. The expression, “a fair chance”, can be interpreted as anything from 0.1 to 0.9 by different people, particularly in different cultures.

Culture also affects a person’s views on probability and likelihood. It has been shown that exposure to games of chance affects a person’s understanding of probability. For some cultures there is no probability. Things will either happen or they won’t, depending on the will of God. To attempt to predict is presumptuous.

These considerations are important when teaching about probability and models. Decision trees rely on good probability estimates, so it is vital that students understand the pitfalls of different sources. Even when we know and understand the rules of probability, we fall victim to the mistakes identified by Tversky and Kahnemann.

You may be interested in my other blog, Never Ordinary, about life with an autistic savant, who REALLY doesn’t like uncertainty.

Please comment on the article here: Learn and Teach Statistics and Operations Research

Dropping out dropouts is a mischievous act

junkcharts — Mon, 07 May 2012 03:34:33 +0000

(This article by junkcharts was originally published at Numbers Rule Your World.)

The following throw-away lines in a Wall Street Journal article about the "return on investment" of getting into college debt (what an idea) are the most important ones:

The report [by the College Board] also doesn't account for dropouts or extra college years. Only 56% of students who enroll in a four-year college earn a bachelor's degree within six years, according to a report last year by the Harvard Graduate School of Education...

PayScale, a Seattle data firm, examines the links between pay and variables like colleges and majors. Its analysis, which also ignores dropouts but accounts for students who take longer to complete their degrees, ...

I cut that off since I've heard enough. How can they get away with ignoring dropouts when they are assessing the return on investment of college debt?

Imagine a cohort of 10,000 students starting college on debt. By year 6, which apparently is when they stop counting, 4,400 have not graduated, either because they dropped out or they are still in school. Both of these groups are likely to have the lowest return on investment of those in the cohort. Most of the dropouts won't be getting college-graduate jobs which pay higher. Those still in school are probably troubled students who if they do graduate later, would also earn less - even if they are equally qualified, they would earn less by value of time.

Given this reality, the analyses by the College Board and by PayScale would "ignore dropouts" as if they didn't ever exist. In other words, they only look at the 5,600 not the 10,000. This means whatever return on investment they compute will be exaggerated.

***

Technically, this is an example of survivorship bias. The sample being studied does not contain "non-survivors", in this case dropouts, so it doesn't generalize properly.

Also, the data is censored in the sense that the observation window is not enough for us to know what would happen to those people who are in college longer than 6 years. This is a common feature of such data sets; you'd want to do something about it, not just ignore it.

There are in fact many other problems with this type of analysis. Here's another crucial one: the counterfactual for reasoning whether debt is the cause of higher future wellbeing is not having debt. In other words, any such analysis must tell us what would happen if the same students were able to complete college without having to incur debt. Based on what the WSJ reporter said, I don't think this is how they framed the problem.

Please comment on the article here: Numbers Rule Your World