The ratings game

It normally takes me nearly a day to write a letter that captures what I know of a person and describes the work he or she has done. Fortunately, it's often possible to use the same letter for every place that someone is applying to, so the effort can be amortized over half a dozen schools.

Some clerical parts of the process have gotten better over the years. I once wrote, addressed and mailed 33 [sic] real physical letters for a friend who was trying to get a teaching job. (He was a great teacher, but times were tough, and he wound up programming at a software company.) Today, physical letters are actively discouraged, and writers are supposed to visit a website, fill in some checkboxes and upload a letter. Commercial operations like Embark and ApplyYourself provide this service for universities; Princeton's graduate school uses Embark. For the first few years, these systems were terribly clumsy and unreliable, but the kinks have gradually been worked out, and mostly things work fine. Some universities and fellowship operations roll their own application systems; those too have gotten better, though they still have bugs. One first-rate university on the West Coast wrote its own system. I have a good friend there, and every year I can pull his chain by pointing out some new glitch or rookie mistake in the user interface, even though none of it is his fault at all.

Perhaps surprisingly, the hard part of recommendations is not the letter itself, but that every single university or fellowship wants something a little bit different from all the others. Some insist on telephone numbers with separators between the digits, while others refuse them. Some require Word documents while others insist on PDF. A few set length limits, so my carefully crafted 800 words have to be chopped and mangled to fit an arbitrary limit like 600.

The thing that irritates me the most, however, is that almost without exception, each place wants the applicant rated against his or her peers on a bunch of dimensions. A typical form lists eight or 10 desirable attributes like creativity, ability in written expression, oral expression, personal character, research ability, teaching ability, and the like. It asks me to assess the applicant on each, against other students, on some non-linear scale like "best ever," "top 1 percent," "top 5 percent," "top 15 percent," down to "bottom 50 percent."

This is ridiculous, to put it mildly. Occasionally there is someone who is clearly the academically strongest this year, but even so, it's impossible to slot that person into "top 5 percent" on "written expression." What could it possibly mean to say that Jack is in the top 25 percent in "integrity"? If Jill is in the top 10 percent in creativity, and Jack is only in the top 20 percent, does that mean that she'll do better than he will? It's not at all clear that these fine distinctions on unquantifiable characteristics have much to do with likely success as a grad student, where perseverance and the ability to sustain or at least feign interest in one's topic for years are more important.

So the natural tendency is to place everyone in one of the top couple of positions, on the theory that anyone merely in the top quartile is doomed. This leads to a Lake Wobegon effect, where all the children are above average. I suspect that the attempt by universities to get a fine-grained and multi-dimensional ranking of their applicants is destined to fail, because most recommenders can't or won't make the distinctions that are asked for.

In a way, this is the same situation that we see every August when US News and World Report publishes its ratings of college and universities. For about eight years in a row, Princeton was number 1 (naturally) but slipped to number 2 last year (clearly some kind of error at US News). Everyone knows that such rankings are meaningless: The criteria can't be quantified, the data is necessarily flaky, and it's all combined with arbitrary weights. It's the same with grad school applications, and pretty much anything else that asks us to rank-order people. There's no way that anyone can put Jack ahead of Jill or vice versa with the precision that's expected in these forms. Let's stick with careful written assessments. People can't be reduced to a single number or even a dozen numbers, and we shouldn't even try.

Brian Kernighan GS ‘69is a professor in the Department of Computer Science and is a Forbes faculty adviser. He can be reached at bwk@princeton.edu.