Comments on: Binomial distributions and multiple choice tests

By: GMP

GMP — Sun, 25 Nov 2012 20:04:10 +0000

I seem to recall that optimum-difficulty tests have the average between 70 and 80%. Anything below would mean the test is too hard, anything above that it’s too easy, and either extremal case comes with poorer correlation between score and knowledge.

By: Lior

Lior — Sat, 27 Oct 2012 17:30:39 +0000

Regardless of the poor results, I think that it’s much better to have the number of choices in the area of 6-8, for noise-cancellation purposes. You can always help the guesswork by including 1-2 options where the units/magnitude don’t fit

By: Craig

Craig — Sat, 27 Oct 2012 16:09:38 +0000

I remember giving a MCT exam to a class of ~100 very weak secondary school students in a physical science class. I remember there being 4 choices for each question. The median grade was, I recall, about 20%. This was rather distressing, as you can imagine, and I vowed not to write such hard tests in the future.

By: Massimo

Massimo — Thu, 25 Oct 2012 14:04:32 +0000

Yeah but come on, now, it is an exam, people are nervous and check the wrong box by accident… plus, are we getting in the business of establishing how much “more wrong” is an answer than another ? And all of this for what, for some contamination from random lucky guesses, whose effect can be rendered negligible ? It’s not worth it.

By: JF

JF — Thu, 25 Oct 2012 07:03:53 +0000

Surely it depends how wrong the wrong answers are ? If you make sure the wrong answers are, say, expressed in inconsistent units, wrong by orders of magnitude, etc., then negative marking is perfectly justified, and will not be affected by simple algebraic mistakes !

By: Massimo

Massimo — Thu, 25 Oct 2012 04:48:02 +0000

But then the exam would not fit in the screen of my mini iPad…

By: transientreporter

transientreporter — Thu, 25 Oct 2012 04:47:07 +0000

Twenty questions? That’s it? You should increase that by an order of magnitude.

By: Massimo

Massimo — Thu, 25 Oct 2012 04:45:47 +0000

Negative marking seems really unfair. There is such a thing as missing the right answer due to an algebraic mistake, and to regard that akin to a blind guess does not seem right (to me).

By: Lior

Lior — Thu, 25 Oct 2012 01:29:18 +0000

Well, the farther away you can get from this randomness in the students’ solution – the better signal-to-noise ratio you can obtain at your measurement. I agree that the deeper you go into the random region, the more unfair is the redistributed curve (if you indeed normalize it somehow). This is why I always prefer to shift the grades by a constant (and not perform non-linear operations) if a correction in the distribution is needed.

The effects discussed here can also occur in non MCTs exams where people try to guess / apply a formula they remember a solution (but it is harder to quantify this effect here).

By: Cath Ennis (@enniscath)

Cath Ennis (@enniscath) — Wed, 24 Oct 2012 22:45:49 +0000

Most of the multiple-choice tests I took during the last couple of years of high school and first couple of years of undergrad – all of which contributed in part to a final grade – featured negative marking. i.e. you get +1 point for answering a question correctly, 0 if you don’t answer it at all, and -1 if you answer incorrectly. It was set up this way to discourage guessing. How would that affect your distribution? (you know you want to run that analysis!)

Many of these tests were also of the kind where each question includes a list of five statements about a single phenomenon, and then the multiple choice part would be phrased something like “which of these five statements are correct? A: #1, 2 and 4; B: #1, 3 and 4; C: #2, 3 and 4; D: #3 and 5; E: #2 and 5″

Are UK multiple choice tests just intrinsically more evil, I wonder?!