One more reason to remove letters of recommendation when evaluating candidates for jobs or scholarships.

This is just one more sexual harassment story, newsworthy only in the man-bites-dog sense. But it reminded me of something that gets discussed from time to time, which is that we should stop using letters of recommendation to evaluate candidates for jobs or scholarships.

Here’s a list of hoops that people recommend you jump through. How horrible!

Here are the usual arguments against requiring letters of recommendation: They discriminate against shy students, they favor students who are bullshitters as well as the students of professors who are willing to hype, they’re used by admissions and hiring committees as a replacement for evaluating the candidates’ actual records, they perpetuate old boys’ networks, they promote flat-out lying (it’s an arms race: if your advisor honestly assesses your strengths and weaknesses, the resulting letter for you can’t compete with the letter from some other prof who is willing to hype), and they’re a waste of time and effort, both for the letter writer and the person who has to track down three people to write the damn letters.

I agree with all the above criticisms. (For elaborations, see for example here, here, and here from Fabio Rojas).

In addition, this latest harassment scandal (also this one) made me particularly aware of another problem with letters of recommendation, which is that they’re an uncontrolled source of power, which creates three problems. First, some people will use this power unethically. Second, students can be in confusion, not knowing if this power will be used against them. And, third, the information in the signal is distorted. We sit in these committees reading these letters of recommendations . . . it’s a joke, it’s a scandal, and it’s part of that horrible power game.

I agree with the commenter who wrote:

I think a letter of reference has two requirements: it should show that the applicant isn’t lying about having been a student of [the recommender], and it should show that [the recommender] would not be positively embarrassed to have helped persuade the hirer to take the applicant on.

That’s fine with me, it’s like what companies do, when someone in the Human Resources office calls your recommender to check that you really existed, that you really worked or studied at the institution in question, and that you showed up on time, did your job, and didn’t break any vases when you were there.

Beyond that, potential hires might want to know what you learned in school, so it would make sense for them to see your grades. Nowadays, we pretty much just give A’s to our Ph.D. students, but that could change. I think it would be fairer all around to give some B’s, C’s, and D’s, and let this convey some information. We could grade the Ph.D. thesis too. They do that in some countries.

This is a basic principle of psychometrics: It’s better to get many pieces of information from different sources. For evaluating recent graduates, better to be evaluated based on a portfolio of grades, than from these letters. For evaluating people who’ve been out of school for awhile, better to be evaluated based on a portfolio of work products than from letters that are possibly rigged and also largely based on out-of-data information.

And, sure, informal recommendations are still gonna happen. If your advisor thinks you’re great, he or she can contact people and tell them. Or if your advisor hates you, he or she can try to blackball you. No way of stopping that. But I think this would all be a lot harder to do, if letters are removed from the formal system of evaluation. I’ve been on a lot of hiring and admissions committees, and people are often using letters of recommendation as definitive evidence. I don’t think this would happen so much if letters no longer had their formal role.

P.S. Here’s another example:

A former assistant reported to a Harvard human resources office in late 2008 that Dr. ** was sending her unwelcome and sexually suggestive nighttime text messages. . . . He refused to write recommendations for economics graduate programs she was applying to, according to her complaint, and all rejected her.

P.P.S. I was corresponding with an economist colleague about the idea of eliminating formal letters of recommendation, and we had the following conversation:

Economist:

I’m not sure what the alternative is, a phone call? A rating system?

AG:

1. Grades
2. Manuscripts

Of course you can’t stop informal recommendations. But I think it would be a step forward to remove the formal recommendations. Now it’s my impression that these recommendations are the first thing that many people read. And I’m getting sick of various stat profs who each year describe their favorite students as the second coming of Jesus H. Cauchy.

Economist:

Won’t work for Econ. Fewer are published at first job market (though that is changing). Also, they want to know what role advisor played in the manuscript.

AG:

Knowing that there are no official recommendations, we can make grades more informative. As for manuscripts, they don’t have to be published to be sent to the committee.

Economist:

But good grades do not imply good researcher, unless you have an informative researcher quality grading system

In the Econ job market people don’t read papers until much later in the process, and they want a signal of the paper’s quality from someone who has their reputation at stake. Maybe they should just ask for that.

Maybe so.