Rantings over rankings

Times Higher Education has just published its influential rankings of World universities. I can almost see university presidents all over the world, at this time, either pounding their chests, proudly announcing to their students that the reputable institution to which their money goes, has climbed from 369th to 347th place in the world, a fact of the utmost importance clearly to ascribe to the vision, hard work and resolve of its administration… or, shrugging off the news (typically reported by the campus paper) that their institution has yet again slipped in the rankings, flatly dismissing the entire operation as petty and insignificant, glorified propaganda, and denouncing the inherent bias, inaccuracy, unfairness and futility of all rankings.

Not surprisingly, individual rhetorics change, sometimes radically, one year to the next, depending on how the rankings themselves evolve.
For example, my own institution is ranked 100th in the world [0], which in and of itself would not seem that bad, were it not because its President (who at this time is siding with the rankings skeptics party), at some point had adopted the now infamous slogan “20 by 20”.
Anyhoozle… Dissecting academic rankings, exposing their capriciousness, inconsequentiality and yes, unfairness, is arguably one of the favourite pastimes of most academics — except those working at highly ranked places. So, friends and colleagues are often surprised to learn that I take rankings of universities, at least some of them and/or some aspects of the whole exercise, somewhat seriously. Eye rolling is typically followed by such remarks as:
“How can you possibly pay attention to such a ridiculous beauty contest ?”
“Can’t you see that they are all about perpetuating the status quo ?”
“How can they even compare different institutions ?”
“Are you going to tell me that rankings reflect a student’s real learning experience ?”

Make no mistake, in most of these cases I agree with my interlocutor, e.g., on the remark that ranking universities is likely not even entirely well-defined an exercise. In fact, some of the most fundamental objections to rankings are not even debatable.
Unfortunately, that is besides the point. Most of the criticisms of rankings of universities that I hear, are not so much flawed, as simply irrelevant. My reply is almost invariably “Yeah, it is true… and your point would be ?
I do not see rankings being outlawed any time soon. Our society likes rankings — no, it is obsessed with them, including those of universities (I am told that US News and World Report’s issue on rankings of universities, is one of their best selling).
Arguing over whether or not people should pay attention to rankings is vacuous. It makes just about as much sense as debating whether Oprah Winfrey’s recommendations should affect the sales of a book or not — they just do, right or wrong as that may be, which is why not many authors would decline an invitation to present their latest novel on her show.
Conversely, telling prospective college or graduate students that the reputation of the institution that they should choose, as assessed by the most popular rankings, will likely have no bearings on their future professional opportunities, is naive at best, downright disingenuous and self-serving at worst.

You mean, you really think that they matter ?

Are you kidding me ?
Of course the reputation of the university matters. Everything else being equal, given a choice between two universities it generally makes sense to opt for the one that enjoys a consistently higher standing (I come back to this below).
That is obviously not to say that one cannot receive an outstanding education even at a second-tier institution, and yes, in some, or even many cases a less prominent university may turn out to be altogether a better choice, just because life is (thankfully) much too complex to reduce everything to a single number.
However, there is no denying it — when it comes to making hiring decisions, most employers do ascribe at least some weight to the perceived reputation of the school where prospective employees graduated, and rankings largely reflect that perception.
Unfortunate and misguided as it may be, this state of affairs is not likely to change in the foreseeable future. Instead of bitching about them, or engaging in some donquixotish battle, I think that it is more sensibly pragmatic to try to adjust to a world where rankings are the norm. Students opting for a lesser-ranked university should do so being aware of the implications. It also means, for those who are in the position to do so (e.g., presidents, provosts, deans and department chairs), taking actions that can improve the standing of their own institutions — within reason, of course.

Are rankings always important ?

In Europe, e.g., in a country like Italy, students normally attend the university that is closest to where they grew up and live. Although there is a sense there too of which universities are more prestigious, in that context rankings can be safely ignored, most of the time.
In North America, on the other hand, students “shop around” for their college education, and are willing to endure both the expenses and inconvenience associated with living away from their parents for a few years, in order to find the university that best suits their need. That means that universities compete with one another, and no institution can take for granted the choice of college bound students living in the area. And even the flagship public institution of a state or province like Alberta must compete locally with other universities and colleges.
But it is at the graduate level where reputation really becomes a factor. A large chunk of the graduate student population at any university consists largely of out-of-state (or, out-of-province) and international students. Those students do care about rankings, primarily those of their department of interest, but also those of the institution as a whole. The truth is, few university presidents who say in public “We do not care about rankings” can truly afford to mean what they say.

Yeah but… which rankings ?

It is true that there are quite a few different rankings, based on different methodologies and metrics [1], and that their assessments of specific universities sometimes differ vastly, especially as one moves past the first 20-30 spots (on those, there seems to be broad agreement among the various rankings, notably the ones mentioned in note [0]). It is probably true what many observe, namely that the top 20 or 30 spots are essentially unattainable for the vast majority of institutions, for a number of reasons (historical, geographical, financial, etc.).
There a lot of institutions, however (like mine), which on the one hand do not seem to have a shot at being top 30 on any ranking (not any time soon anyway), but at the same time manage to be featured among the first 100-150 on most (if not all) of them.
If I were a prospective college or graduate student, or a parent, I would list among my selection criteria whether or not a university is consistently ranked in the top, say, 200 in the world. If in doubt between two or more choices, I would give my preference to the school that is consistently ranked above the others [2].
If I were a high-rank (no pun intended) university administrator at one of these schools, I would value this kind of consistency, and without obsessing over it I would take action to ensure that the placement of my institution across the various rankings improve consistently, or at least remain at its current, acceptable level.

Actions like… what ?

There is no question that financial resources play a huge role in this business. If resources are not there, if support from the state or the province is not on par with that enjoyed by competing institutions, if no wealthy donors show up (sometimes they do, and have a tremendous impact — see here, for instance), then it is always going to be an uphill battle. Funding, money, does play into the criteria that affect a university’s standing, which are essentially the same for all of the various rankings — for example, “Student to Faculty ratio” is clearly dependent on the size of the full time teaching staff that the university can afford. That is why fund raising is (or, should be) a major part of the activity of a university president.
At the same time, if one looks at other widely adopted measures, one finds things like “international character of the faculty and/or student body”, and it seems to me that there are strategies that universities could adopt to improve on those fronts without incurring into massive expenditures [3]. Obviously, imposing large differential tuition and fees on international or out-of-state students, putting the institution at a competitive recruiting disadvantage, will not help to that effect.

Scholarly Research

There may be disagreement on how to evaluate it, but that scholarly research is a major indicator of the success of an institution of higher education is not really up for discussion. A university, especially a research one, is expected to be a centre of vibrant intellectual activity, a laboratory of ideas, contributing to extending the boundary of human knowledge, generating innovation in science and engineering, as well as in the arts and humanities, ultimately promoting and inspiring technological and social progress. And it is mostly through research that a highly qualified workforce is educated and trained.
Now, “scholarly reputation” is something that is acquired over the course of decades, and there is little that one can do to improve it, other than by trying to recruit effective faculty — which of course requires a financial investment, and whose payoff may not be immediate.
On the other hand, one can try and improve upon quantitative measures such as “citations per faculty”, mainly by pursuing high tenure standards, but also by concurrently ensuring that the number of faculty who are mostly engaged in teaching and/or academic service (i.e., not in scholarly research) be kept down to a minimum [4].


[0] Interestingly, this is remarkably consistent with the evaluation of University of Alberta offered by the QS ranking (in fact, identical with this one) as well as by the Academic Ranking of World Universities (ARWU). On the other hand, according to US News and World Report, University of Alberta is the 78th best university in the world. Well, I guess we now know whose methodology is clearly most reliable and accurate…

[1] This is what renders rankings based on “reputation” alone much less whimsical and arbitrary than one might think at first. For one thing, I do not believe that rankings based on reputation are significantly at variance with the supposedly “objective” ones, based on metrics.
Moreover, it should be noted that rankings based on reputation reflect the perception of society, including of the people in charge of hiring or funding decisions. That is really what ultimately affects the lives and careers of most of us.

[2] Please note my emphasis on the adverb “consistently” — if there are large fluctuations, as a result of which no clear difference can be discerned, it is obviously pointless to obsess over this.

[3] I am no administrator and am not aware of what all of that really entails, but it seems to me that one could start by encouraging and rewarding faculty activities aimed at establishing international liaisons, that eventually might lead to increased recruiting of students and faculty abroad.

[4] Two side remarks: because faculty whose scholarly output fail to generate a decent number of citations (i.e., their work has little impact on their field of inquiry, either because they are not very productive or because their contribution is not deemed sufficiently original) have a negative effect on the “citation per faculty” index, universities and departments should be wary of listing too many such individuals as “faculty”. This consideration should play a role in decisions of granting, for example, “adjunct” type appointments, as adjunct faculty are often included in the citation count.
Secondly, because teaching is obviously a fundamental aspect of the profession, it is important that the overall teaching load be as much as possible spread equally among research active faculty. For, overburdening any individual professor with an excessive teaching assignment will likely have the effect of weakening, if not downright crippling, his/her research program. By the same token, faculty who for whatever reason are no longer active in research and/or supervising graduate students should pick up a greater share of the teaching.
None of that requires squeezing extra monies from the legislature.

14 Responses to “Rantings over rankings”

  1. Schlupp Says:

    What I really do not understand is the following: As soon as I say that I think it a bit dumb to pretend that only people from the top place (Cal Tech in the present case) are worthy of any respect, people act as if this implied that I believe all places to be the same.

    Ah, and dear European beginning grad students: Yes, you can stay at the place where you were born for grad school. Yes, this may be the best choice for you personally, depending on many, many different things. I’d just like to point out that it *is* a choice.

    (Well, yeah, granted, what I also want to say is: I’d make a better adviser than that nice guy who can unfortunately not pay you because funding agencies are so mean and whose students never publish paper, because his standards are so much higher than the journals’.)

    • Massimo Says:

      As soon as I say that I think it a bit dumb to pretend that only people from the top place (Cal Tech in the present case) are worthy of any respect, people act as if this implied that I believe all places to be the same.

      Yes, I think it must be the same people who, when I say that we should pay some attention to numerical indicators, tell me “Oh, come on, we cannot judge applicants only, exclusively based on numerical indicators, as you are clearly implying ! “.
      I think it is technically called a “straw man argument”.
      In general, what they really mean is “Now, now, why must you confuse the issue by mentioning criteria that penalize my favourite candidate, namely the person who will get the job in the end anyway ?”.

      I’d just like to point out that it *is* a choice.

      A bad one, in my opinion. Unless you know for a fact that you will get a job at that very place (and I would tend to find that either naive or objectionable), I would not recommend that path to anyone.

  2. GMP Says:

    I know it’s lame to leave a comment stating “I agree 100% with what you wrote,” but here I am, doing so anyway.

    Btw, don’t you love it when people who hate the h-index say something like “But Einstein’s h-index is like 5 (or whatever, something small)” which is supposed to show that the index is meaningless. I want to tell them “Dude, you are sooo not Einstein. There is a much simpler explanation for your low h-index.”

    Metrics are imperfect, but they are quantitative and we can all understand what it is that they measure. They are a bite-sized quality assessment, and that’s what makes them so popular. They are not going anywhere, ignore them at your peril.

    Btw, my university is in top 30 in the Times ranking so I am feeling all smug now. πŸ˜‰

    • Massimo Says:

      Btw, my university is in top 30 in the Times ranking so I am feeling all smug now

      I so wish I could stick my tongue out at you now, but it’s OK — my President is doing it for all of us πŸ˜€

      Oh, and regarding the “you are no Albert Einstein” comment here is how it is done.

  3. Mac Daddy Says:

    4th in Canada!!!!111!!1!

  4. Transient Reporter Says:

    telling prospective… students that the reputation of the institution that they should choose… will likely have no bearings on their future professional opportunities, is naive at best, downright disingenuous and self-serving at worst…

    Since we’re all unzipping ourselves and showing off our universities, I just wanted to let you know that I attended a Big Famous University as an undergrad (in the Top Ten of that little world ranking thing you’ve got there). I was taught by Big Famous Professors with GIGANTIC egos who couldn’t care less about their students. After I graduated, I went to get a job and my prospective employer said, “Fuck you, I ain’t hiring one of you Big Famous University boys with your big egos.” I finally got a job somewhere, and went to ask a colleague for a favor, and the response was, “Fuck you, I ain’t doing any favors for you, Mr. Big Famous University.”

    • Massimo Says:

      I am making an exception to my blog rule allowing this comment, and I do it ONLY because you are a BIG FAMOUS university graduate — otherwise I would have gone “oh, #%^* you !” and hit the “spam” button.

    • Transient Reporter Says:

      To what extent does attending Big Famous University limit your options? Of all my college friends – the vast majority of them (the vast, VAST majority) are doctors, lawyers, businesspeople. And by “businesspeople” I mean, Wall Street lackeys running around doing deals. A tiny fraction of us went on to do other things (no doubt this has something to do with the kinds of people I hung out with – i.e. people who liked to get drunk).

      But is Big Famous University limiting, in the sense that if you are a graduate of BFU, you will naturally choose career paths that will especially value the BFU cachet – not to mention justify the debt you accrue.

      Is a BFU degree a gilded cage?

      • Massimo Says:

        Well, I don’t know what to tell you, my friend, you are asking deep philosophical questions now… All I know (because there used to be data on the internet, on a blog that is now no more) is that the vast majority of faculty positions in North America go to PhD graduates of the top 10 institutions.
        The cause/effect relationship is not obvious, and he significance of that is for each and everyone of us to assess. I did not go to a BFU and look how awesome I turned out…

      • GMP Says:

        that the vast majority of faculty positions in North America go to PhD graduates of the top 10 institutions.

        One can quasi-prove this quite easily, even if the data is no longer up on that blog: it’s enough for a person to just go to any major research university website in any discipline and look where their profs got their PhDs. For each discipline there’s about 10-ish schools whose graduates populate 80+% of all faculty positions. What those 10 schools are varies a little with discipline, but for a given discipline they are quite easy to identify.

  5. Cath@VWXYNot? Says:

    Damn, Glasgow (where I did my PhD) came in 102nd, so I can’t use your arguments to gloat πŸ™‚

    • Massimo Says:

      Well, Florida State, where I did my PhD, apparently has been ejected from the club altogether, given that in most of those rankings it is not even top 400 …not quite sure what to make of that, did not seem so awful back in my days… then again, I did leave, and you know, good luck recovering from that

      • Schlupp Says:

        OK, I can join in that game: My university was happy when they are were permitted to take part in a ranking of the world’s *worst* universities. And that was before they lost me. In the snow and uphill.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: