(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
This post is by Phil. It is not by Andrew.
I’m halfway through my third year as a consultant, after 25 years at a government research lab, and I just had a miserable five weeks finishing a project. The end product was fine — actually really good — but the process was horrible and I put in much more time than I had anticipated. I definitely do not want to experience anything like that again, so I’ve been thinking about what went wrong and what I should do differently in the future. It occurred to me that other people might also learn from my mistakes, so here’s my story.
The job was a ‘program review’ for a large, heavily regulated company that has an incentive program to try to get customers to behave in certain ways that both the company and its regulators feel are desirable. (Sorry, I have to remain pretty vague about this). The company is required to have an independent contractor review the program annually, to quantify the effectiveness of it and to predict how the program would perform under a few hypothetical futures. The review had been performed by the same consulting company for several years in a row, and it looked like they had done a competent job, so my partner and I felt that we were at a substantial disadvantage bidding against them: the other company already had experience with the data and the process, they had computer code already written and personnel who knew how to run it, and they would presumably do most of the work with junior staff whereas my partner and I are both quite experienced and not willing to work for the low wages a junior person would be happy with. Still, we saw several places where we were sure we could improve on the previous work, and we had been told that the company likes to switch consultants every now and then, so we put in a bid, and we got the job.
An initial stage of statistical manipulation was needed in order to process the raw data into numbers that would feed into the statistical model that would spit out the outputs, and it’s in that initial stage that we made our improvements. For the model itself, which was in my bailiwick, I was going to use pretty much the same model that had been used in the past. The previous reports noted several potential problems with the model, and said they had checked and the effect of the problems was small. There was one thing about the model I really didn’t like: the use of two very highly correlated explanatory variables in a linear regression model. That’s OK if you are making predictions for regions of parameter space that are well sampled in your training data, but can be a big problem when extrapolating, which is what we needed to do. So I changed the variables in the model to use a less highly correlated set, checked that the model still fit about the same, and kept plugging along.
So everything was going fine…right up until the first major ‘deliverable’, about five weeks ago. All the client needed was a few hundred numbers in a table, representing the estimated performance of the program under those hypothetical futures I mentioned earlier. The numbers were due on a Monday, and I had them ready on Friday: I was looking forward to a weekend getaway with my wife and some friends, and didn’t want to have any nagging worry about not being able to submit the numbers on time. So, Friday I sent the numbers to our project manager at the company…and got back an email that afternoon that said: why are the numbers so low? Last year they predicted such-and-such, and you guys are 20% under!
My first thought was: uh oh, where did I screw up? Each of the final numbers is the sum of several other numbers, maybe I forgot to add one of them? Or maybe the problem is in the pre-processing pipeline, which would cause a problem with the inputs to the model? Or maybe…
I worked through all of Friday night, checking one thing at a time, but I didn’t find anything wrong. When my wife woke up Saturday morning I had to send her off with our friends on their getaway; I kept working. I worked all Saturday and still couldn’t find the problem; ditto Sunday. Sunday afternoon I finally roped my partner in, and we spent the day cleaning things up in my analysis code and checking as much as we could check. And we did find some things wrong! I had failed to adequately specify merge parameters in R’s ‘merge’ function, which was causing some numbers to end up in the wrong places; the client had sent us some corrections to their data but those hadn’t made it into our database; and a few other things. None of these turned out to make much difference, but the fact that we were finding little errors made it seem possible that there was one (or more!) big one out there. And it was frustratingly hard to check things or test things because I continue to use some sloppy programming practices that I swore a year ago I would improve upon. Finally Monday afternoon rolled around. We had checked every module and every bit of code, we could not find any problems at all anymore…and we were still generating numbers that were 20% under what previous years had reported. We had no choice but to submit our numbers just before the deadline, with a nagging worry that there was still a major problem. The numbers were due at 5 pm Monday, so we submitted them then, but we asked “obviously nobody is going to start using these tonight and there are still things we would like to check out; when is the time people actually start with these?” The answer: start of business Tuesday, that’s why they were due Monday. So we kept working Monday evening after submitting the numbers.
Through all of this, my assessment of the situation had changed. On Friday, when first told that our numbers were different by 20%, I would have said there was a 90% chance the problem was my fault. Saturday night, maybe I would have said 60%. Sunday night, 30%. Monday evening, maybe 10%. So I was 90% sure we were OK, but that still meant a 10% chance that we weren’t!
Then, around midnight Monday night, my partner and I were looking everything over once again. I was looking at the previous year’s report, and I said “I just don’t understand how they could possibly get the numbers in the second column of such-and-such a table, these completely disagree with us and they don’t even seem plausible.” And then I noticed a footnote to the table: “Numbers in the second column have been adjusted under the assumption that…” And suddenly it all fell into place. The assumption was wrong, indeed easily demonstrated to be wrong, and it was wrong enough that replacing it with the right answer made about a 20% difference. My partner and I went to bed. I got a full night’s sleep for the first time in days.
So was that three days the visit to Hell I alluded to? Oh not, that was just the start. First, we faced some pushback from the client. We were telling them (and ultimately the regulators) that the program wasless effective than they thought. They were reluctant to believe it, although (to my relief) they were not inclined to shoot the messenger. Still, for the next couple of days I did little but try to document the evidence that the previous report had gotten it wrong. So all in all this little episode cost about a week. When you’ve only got five weeks to write the final report, and you spend one of them on something you didn’t anticipate, the pressure mounts. Still not Hell, though.
The real problem was: having realized that the previous work had involved this major misjudgment, I no longer trusted that work in other ways..specifically, I no longer took it for granted that the statistical model that had been used in the past was adequate. I finally did what I should have done months (!) earlier: rather than make a few plots and tabulate a few things to confirm that the model was behaving OK, I started looking for evidence that the model had significant problems. And sure enough: the model had significant problems. I spent several solid days coming up with a final model that I was willing to stand behind…which doesn’t sound so bad, but significantly expanded the workload: now, instead of just saying “we used the same model they used last year, except for such-and-such a minor modification”, we needed to explain why we did what we did, quantify how much difference this made, and so on. Of course this meant that the numbers we had submitted earlier — a week and a half ago, at this point — needed to be changed, because we have this new model, so that was a slightly embarrassing issue with the client. And we eneeded more tables, more plots, a section comparing the models and explaining the differences that are due to the model (in addition to a section explaining what had changed about the program; these two now had to be untangled). All of this with the clock running. Having lost a week diagnosing the problem with the previous model and convincing the client that it was indeed a big problem, and then another coming up with an improved model and figuring out the implications, we now had about three weeks to write the report. Five had already seemed pretty tight, to write what we expected to be a simpler report. Having 3/5 of the time to write a report that was 5/3 as complicated (or something)…well, we got it done, and I am pretty proud of it, but the five weeks from the start of this narrative to the moment we finished were as unpleasant a period of work as I’ve experienced in 25 years.
What are the lessons I am taking away from this?
The biggest lesson is: You know your model is not perfect, which means you know there are ways in which the answers it generates are wrong. You need to know what those ways are, and see whether the results are good enough for your purposes. If they aren’t, you need to modify the model. My initial mindset was more like “this is the model that was used in the past, let me check a few things and see if it makes sense”, and that’s completely wrong. My goal should have been to try to demonstrate that the model was inadequate, not to try to check whether it’s adequate.
The second lesson is: I really do need to code better. I did not find any really major errors in the work that I had done to generate the first set of numbers, but I could have; indeed, it seemed so plausible that I had made a major mistake that I assumed that must be the problem. Right now I am very sure that my analysis, and the code that implements it, does not have any mistakes of practical significance, but that was definitely not true at the time that we submitted our first results, and it should have been. This is partly a matter of simply allowing more time, but there’s a big component of better practices: create modules that can be tested independently, and then create tests for them, those are two of the big things I didn’t do initially.
Finally — and this is a minor one compared to the others — I should have been a lot less trusting of the previous work. In fact there were some major red flags. I already mentioned the use of two highly correlated variables in a regression, with no attempt to ensure that they could reasonably be jointly used to extrapolate beyond the range of the data, nor even any discussion of the issue. Also (perhaps related to that one) I had of course noticed that the previous analysts had not tabulated the regression coefficients of their model. I assumed that was just because the intended audience didn’t care about, or wouldn’t understand, such a table…but shouldn’t they have put it in anyway? (In fact, it is a requirement by the regulators, had anyone chosen to check). In essence, I had made the mistake of judging the book by its cover: I assumed that since the report appeared to have been done reasonably well, it actually had been.
I may not be able to eliminate these mistakes completely — especially the coding one — but even half-measures there are better than no measures at all.
Do as I say, don’t do as I did: (1) Always try to find all of the ways your model is significantly wrong, and to understand the magnitude of its deficiencies. (2) Implement the model in code that can be tested one module at a time. And (3) if someone tells you some specific model is good enough, don’t believe them until you’ve tried to prove them wrong…really this is just a corollary of #1.
Good luck out there!
The post Lessons learned in Hell appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
