You’ll recall that yesterday, in reference to the orgy of hype that accompanies the annual release of the THE World Reputation Rankings, I made the point that universities’ reputation really doesn’t change all that much on a year-by-year basis, and that, therefore, perhaps said orgy was a wee bit overdone.

When all was said and done, five universities (Arizona, Indiana, Leeds, U Zurich, and Tel Aviv) fell out of the rankings, with a similar number replacing them (Monash, Moscow State, Freie Universitat Berlin, New South Wales, and Maryland).  Now, this might sound eminently stable – a 95% recidivism rate!  But actually, having five positions in the top 100 shift in a single year is actually relatively volatile.  What explains this?

Well, how about a change in the geographic balance of respondents?   Unlike baseball’s all-star ballot, the THE rankings doesn’t discount voting for home-town favourites.  And yes, stuff like that is happening. How else can you explain Turkey’s Middle East Technical University regularly cracking the THE’s top 100 when it doesn’t even make the top 500 in the ARWU?

THE isn’t consistent about the way it reports respondent populations, but here’s my attempt to compare the 2012 and 2013 samples:

Now, I’m sorry, but when 10% of respondents in a sample – which is meant to be globally representative – are from Australia and New Zealand (possibly more, depending on how that 12% “unspecified” pans out), you have a skewed sample.  It beggars belief to know this and still claim with a straight face that Australia’s moving from 4 to 6 institutions in the top 100 actually means something.  The least they could do is weight the results.

One final treat before I leave this topic for a few months. The keen-eyed among you may have noticed that the THE Reputation Ranking is based on two elements: a “research ranking” and a “teaching ranking”.  The even keener-eyed among you may have read the fine print to learn that, in fact, the teaching reputation is a misnomer: it’s actually a ranking of “graduate teaching”, which you’d have to figure is so correlated to research as to makes no odds – and this is, in fact, the case.  I plotted  each institutions’ score for “Teaching” and “Research” Reputation on a scatterplot, and here’s what I found:

Yes, you’re reading that right.  An R-square of .99.  Kind of makes you wonder why they bother, doesn’t it?