Evaluate me

I like his hair
(from one of my student evaluation forms)

Student evaluation of instruction (SEI) is a frequently discussed subject among academics, one of those that tend to elicit heated debates and strong emotions. This post at academics_anon and the ensuing comments, have rehashed some of the most common aspects of SEI that college teachers find objectionable.
Having taught at the college level for twelve years, I share some of the concerns that a good fraction of my colleagues voice; however, my main problem with student evaluations does not really have much to do with what students write about me. It is just that I have never found the type of feedback provided by students on anonymous evaluation forms all that useful, which in turn makes it difficult for me to take SEI seriously, as a way of assessing teaching effectiveness.

I am not going to discuss here the wealth of literature and research on SEI (for those who are interested, a good place to start is here); instead, I am going to describe my own experience, and make some personal remarks not based on any scientific study.
First off, a full disclosure of my own record is in order. My subject is physics (not exactly a popular one), and over the past twelve years I have been evaluated by students at all levels (and of all levels). Much to my embarrassment, my evaluation record is pretty boring. Aside from the numbers (I typically get averages around 3.5 out of 5.0 at the undergraduate [lower division], and 4.5 out of 5.0 at the undergraduate [upper division] and graduate level), which pretty much put me right there with the vast majority of my peers, countless conversations with colleagues have led me to believe that my evaluations are fairly ordinary.
Obviously, I have received my fair share of, er, “candid” comments (the best one to date remains Good bye – good luck – get high – get … OK, I’ll leave it up to those who are more poetically inclined to guess its ending). In twelve years, only two or three times have I been genuinely hurt by what I read, which I thought was gratuitous, unfair and untrue. On a few occasions, on the other hand, I received comments that were glowingly positive, and that made me feel good about myself, for a little while. For the most part, however, it was all pretty forgettable. These days, I quickly glimpse at my forms, looking for anything unusual and finding instead that they almost look identical to those from the previous year.

I do not mind being evaluated — after all, who the hell am I not to be. I think it is fair to let students voice their opinion, and even do some venting if necessary. I can see how SEI may be sometimes the only way to spot infrequent, but occurring, seriously deficient teaching performances [0]. However, relying exclusively on SEI in order to assess teaching effectiveness is ill-advised.

SEI is often predicated as a way to help faculty improve their teaching, based on the comments made by the students. Personally, I can honestly state that I have not made a single change to my teaching style as a result of feedback that I received from students by way of anonymous evaluation forms. Suggestions from senior colleagues have proven much more useful. It is not that I am unwilling to take advice from students, it is just that I often find it difficult to decipher what evaluations are really telling me, much less act upon it. Student comments are typically ambiguous, inconsistent and whimsical (I suspect this to be generally true of any customer feedback — I would not be surprised if much of it were dismissed as unreliable).

For a physics professor, evaluations are normally important in connection with lower division service courses (i.e., first and second year undergraduate level). When dealing with physics majors or graduate students (i.e., fellow geeks), evaluations are almost like patting ourselves on the back.
Now, whenever I have taught a large (50+ students) introductory physics class (typically for engineers or life science majors) I have consistently observed the following:
1) An instructor is criticized for things that are beyond his/her control. Examples include textbook, course content, make-up exam and course withdrawal policies, as well as the fact that the course is required. Individual instructors have no responsibility in any of that, yet students will put the blame on them, as the persons with whom they are dealing on a daily basis.
2) Almost invariably the most vitriolic comments are directed at testing and grading. Naturally, these are two aspects on which the instructor exercises some (not full) control, but are only indirectly part of classroom teaching.
3) Teaching effectiveness, namely the area in which student feedback is presumably sought most avidly by universities, is one on which students will comment only rarely, and when they do it is very difficult to extract signal from noise. Very often, the same aspect of one’s teaching is praised by some and criticized by roughly as many others. Some students like in-class demonstrations, others think they are a waste of time; some want essay-type tests, others want multiple-choice; some like to see derivations of formulae, others just want more emphasis placed on problem solving, and so on, endlessly. These are all legitimate opinions, of course, but there is no reason for an instructor to act upon a recommendation which only a minority of students would endorse.

The simple truth is, instructors are damned if they do, and damned if they don’t [1]. Obviously, students deserve no blame for doing what they are asked to do, namely express an opinion on the instructor. And none of this would be a problem, if SEI were taken for what they are, namely a way to exercise some quality control, and spot those few, indisputable instances of unprofessionalism and/or incompetence. Anything beyond that is unrealistic. No person or business can expect not to receive a single complaint ever; there is no such thing as “100% customer satisfaction”, and one person’s “best thing after sliced bread” will always be someone else’s “horrible experience”. Only when dissatisfied customers represent a substantial fraction, and/or consistently voice the same concern, is any action really required.
What worries me and many a colleague of mine, is not the notion of being evaluated by students, but rather how SEI will be used by college administrators. One of the most irritating and frustrating experiences consists of being told: As long as even a single student complains, you have work to do to improve, as it is your job to make as many as possible of your “customers” happy. What is that supposed to mean ? It is difficult for a college teacher not to feel, at some point in his/her career, that student evaluations are utilized in a way that is dishonest, or unfair, for the purpose of exercising power over him/her.

Good teaching
How else should a teacher be evaluated if not through SEI, is a question often asked by many a SEI proponent. I do not have a good answer to that, and I suspect that it is probably a good idea to make SEI at least part of the process of evaluation of teaching. But I am nervous about it being the whole thing. Teaching can not be equated to a mere delivering of a good, for which “customer satisfaction” is the only criterion to judge effectiveness. If anything, a more broadly encompassing definition of “customers” is needed, to include not just the student, but also parents, employers (both in the private and public sectors), society at large, whose expectation is that graduates of the institution will be competent.
In many respect, the comparison of a teacher with a physician seems appropriate. Patients certainly appreciate good bedside manners, a friendly demeanor, and good communication skills; ultimately, however, what they really expect of physicians is that they will cure them.
And, curing someone is not the same as making that person think that (s)he was cured
That is really the only thing that matters in the end, and therefore a physician should be primarily evaluated through this lens. And while people skills are certainly part of the package, no sane hospital administrator would ever make the sole criterion to determine whether a doctor is performing adequately or not whether patients walk out of the hospital wishing to go on a date with that charming physician, or in any case feeling good about themselves regardless of any objective assessment of their health.

Now, teachers have an advantage over physicians, in that they can give an illusion of competence through grade inflation, whereas physicians cannot really give patients an illusion of well-being (not a long lasting one anyway). Eventually, however, even students will come to a crossroad where their preparedness will be dispassionately, brutally assessed. Those in charge of doing it will care very little about student evaluations. An employer who comes to the realization that a freshly hired engineer is not adequately versed in electricity and magnetism (E&M), is likely to come to the conclusion that the school where that person studied is mediocre, and that no more of its graduates should be hired, regardless of how popular with students its E&M teachers are. Likewise, to a biology major taking the MCAT, what (s)he thinks of his/her physics college teacher matters very little for her future — it’s the physics that (s)he knows the only thing that does. I am not sure that the two things are necessarily correlated, and in any case it seems to me that the degree of learning should be assessed independently of SEI [2], with an eye to standards that society has set, regardless of what each and everyone of us may think of so and so teacher.

Which is to say that, in the end, the “better” teacher is the one whose students, on average, learn more. Duh ?

[0] Increasingly I hear of universities that actively try to compel students to evaluate their instructors, to the point of withdrawing grades of those who fail to comply. I find this really bizarre. Why force someone who has ostensibly no opinion about something to express one anyway ? The argument put forth by those who spouse the notion of mandatory evaluation, is often that of “customer care”, but I cannot think of any business withdrawing the service from a paying customer in exchange for an evaluation of the service itself. How would you like it if one of these days, the nurse at the hospital should tell you that you won’t get your stitches removed until you have filled out a surgeon evaluation form ? …
[1] Anecdote time: at the end of a term, I received once quite a few comments from students lamenting the fact that other students would constantly walk in and out of the lecture room during class, distracting everyone else. Even though that was in no way my responsibility, and therefore I should simply have done nothing, because I was young and naive I decided to do something about it. So, I adopted in the following term a closed-door policy, i.e., nobody would be let in after ten minutes from the beginning of the class. Aside from bitter complaints that I received during the entire semester, at the end I got a lot of negative comments from students, calling such a policy unfair.
[2] Comparing the performance on a course final exam, or on a standardized subject test, of students who were taught by different teachers, might be a way to assess teacher effectiveness.

Tags: , ,

6 Responses to “Evaluate me”

  1. Anonymous Says:

    Good analysis

    I like your opening quote, too.

    I would really like some good ideas, though, as to how to get some sort of feedback to instructors such that they can learn what they do and don’t do right. Nothing reproduces a course in front of the students as well as that self-same course; practice sessions in front of other instructors would not have the same feel, and often it’s necessary to see teaching behavior over a whole term to see what is or isn’t effective. Only the students themselves get that exposure.

    The reason I am asking is that university instructors in France don’t seem to get any help at all in how they teach, and indeed have no incentives to improve whatsoever. So there are “naturals” and the others. In the current political context of university overhaul I think this would be really useful.

    • Massimo (formerly known as Okham) Says:

      Re: Good analysis

      I wish I had some useful idea… alas, I do not. The thing about student evaluations is that it is really hard to get them to agree on just about anything, except for the most egregious offenses (e.g., a professor skipping classes or chronically late to class, unable to answer simple questions on course material, etc.). But thankfully there are relatively few cases of that type… most of us fall right “under the curve”, and it is very hard to improve if you are there because, again, one is almost inevitably going to satisfy some and disappoint others.

  2. Anonymous Says:

    Hmm…would it be any consolation to find out that students find SEIs to be as useless as instructors?! 🙂

    I agree SEIs are not effective as the sole means of evaluating one’s teaching skills. But there are probably a few changes which could be made that would make them more useful.

    First, as you pointed out, it is ridiculous to force students to write evaluations. This results in students writing a few sentences of bullshit just so they can get their graded exam back, receive a grade for the class, etc.

    Also, having students write evaluations at the end of the course, while logical, may not actually be very effective. Students who have legitimate, and perhaps even useful, comments have no incentive to communicate them coherently because it will make no difference to them! I suspect students may take SEIs more seriously if they believed their feedback might result in changes which would actually ameliorate their “classroom experience.”

    But really, the truth of the matter is that most teachers are average, which is why most teachers receive average evaluations. I can count on one hand the teachers I’ve had who were truly memorable, either for their skill or lack thereof….

    Mad Hatter

    • Massimo (formerly known as Okham) Says:

      Hmm…would it be any consolation to find out that students find SEIs to be as useless as instructors?! 🙂

      Oh, I know, they told me many times… they feel jerked around, as they fail to see any change in the teaching assignments (nor can there be any in most cases, given the chronic scarcity of teaching personnel in many departments). Which of course brings up another point: what is the point of SEI if departments and universities are not ready to implement any suggested change anyway ?

      Also, having students write evaluations at the end of the course, while logical, may not actually be very effective.

      I am not sure I agree. If I am dissatisfied, I am going to want to vent even if it won’t make any difference to me, if anything to make sure that the service improves. In fact, i have always thought that students should be allowed to submit their evaluations only a few months after the end of the term, in order to give them enough time to “cool down” and put things in perspective. I know I would have submitted my evaluations even years after taking a particular class (if there had been any SEI at italian universities in the early 80s, that is…).

      I can count on one hand the teachers I’ve had who were truly memorable, either for their skill or lack thereof….

      Well… but, aren’t most people “just average” at what they do, regardless of what it is ? I know I am (except for looks — I am definitely off the charts there).

      • Anonymous Says:

        what is the point of SEI if departments and universities are not ready to implement any suggested change anyway?

        So the departments and universities can claim they are trying and blame any lack of progress on (1) instructors not making the right changes and (2) students not writing useful evaluations. Classic CYA.

        except for looks — I am definitely off the charts there

        Umm…off the charts in which direction?!! 🙂

        And “italian universities in the early 80s“? I so badly want to make an age-related joke. But I’m going to be nice. For now…. 🙂

        Mad Hatter

      • Massimo (formerly known as Okham) Says:

        age-related jokes are mean…

        My DOB is on my main page anyway… yes, I am an old man (handsome though)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: