Ranking Surveys Typically Are Subjective and Have Unintended Consequences.

Ranking Schemes

by Elaine Schwartz    •    Nov 17, 2012    •    555 Views

I just looked at the U.S. News and World Report ranking for business schools. Then, while I was there, I could not resist checking out the “best” colleges and their new best connected category. The top 3 colleges are Harvard amd Princeton tied for #1 and then Yale #3.The top 3 graduate business schools are Harvard and Stanford, a #1 tie and then Wharton, #3. The best connected list (for example–who has the most wifi coverage) starts with Bowdoin then University of La Verne and Auburn University.

But I kept asking myself…Why?? Whether looking at colleges or income or even GDP, ranking is totally subjective. It all depends on the variables you select, their weight, and how you quantify them.

In an especially excellent 2005 Atlantic article, former Reed College president Colin Diver, detailed the impact of the U.S. News results. Summarizing the downside of ranking while explaining why Reed refused to participate in the process, he took me to a slew of unintended consequences that included an “irresistible pressure toward homogeneity” and the temptation to elevate scores by changing institutional procedures. Meanwhile, he marveled at the quantity of brochures he received from schools hoping to influence the reputational survey he completed annually when he did participate as the Dean of Penn Law.

By contrast, other critics looked at the surveys themselves. In a 2011 New Yorker article, journalist Malcolm Gladwell showed how survey questions reflect the perspective of the designer and skew the results. Focusing on graduate business school ranking by 5 magazines, one Forbes writer perfectly illustrated how different questions generate varied results. In one example he explains that Forbes uses the financial gain of graduates after 5 years as a ranking variable for graduate business schools while U.S. News uses starting salaries. And yet both say they are determining which business school is best.

You probably know where this is going. Ranking schemes exist far beyond the university world. Whether comparing incomes or GDPs, we tend to select variables that reflect our own bias.

Sources and Resources: If you are going to look at one survey, I recommend this new Bloomberg business school ranking because their chart, here, identifies and ranks each variable. The article that was best, though, was the Colin Diver discussion in the Atlantic but always, Malcolm Gladwell is a pleasure and the Forbes comparison of ranking by 5 popular magazines was enlightening. And here are the U.S. News ranks.

Leave a Reply

Your email address will not be published. Required fields are marked *

« »