Blind justice: rebalancing the weighing scales
There is some scepticism as to whether multiple-choice questions are useful. A while ago, I shared this scepticism. I assumed they weren’t very rigorous.
Now that I’ve been using multiple-choice questions for some time, inspired by Daisy Christodoulou, I’m beginning to realise why they are so useful. Daisy argues that they can require higher order thinking, if the distractors home in on important, frequent misconceptions. Daisy went through all of the literature and history multiple choice questions on British Columbia leaving exam, and says ‘all of them made me think hard about deep issues.’ This fits with Professor Rob Coe’s simple theory of learning: learning happens when you have to think hard about subject content. Phil Stock has also come on a similar journey to me in his thinking on multiple-choice questions.
Here are some of the reasons why I think they can benefit teaching:
1. They make assessment more reliable
2. They make marking far less labour-intensive
3. They make pupil understanding more visible to teachers
Let’s take each in turn.
1. They make assessment more reliable
Reliability is best understood with the weighing scale analogy: the chance that if the object has not changed weight, the weighing scale will tell us the same weight. Similarly, you want different assessors to give the same answer the same score.
Extended essays are easy to set but hard to assess reliably. Anyone who has been in an English or humanities moderation meeting, with different teachers giving the same essay different scores on the same rubric, will know that.
Multiple-choice questions are hard to set but are always reliable to assess. My initial assumption that they weren’t rigorous was wrong: it depends on how they are set.
Paul Bambrick Santoyo makes the following case:
▪ In an open-ended question, the rubric defines the rigour
▪ In a multiple choice question, the options define the rigour
▪ Effective assessment combines them: both are useful
As a demonstration, take two example option sets:
When did Charles Dickens write Oliver Twist?
a). 1537
b). 1637
c). 1737
d). 1837
e). 1937
or
a). 1835
b). 1836
c). 1837
d). 1838
e). 1839
The closer the options, the tougher the question. The first option set is not very rigorous; all you have to know is that Dickens wrote in the 19th century. The second is more rigorous; you have to know the exact year it was written*.
Of course, the nature of the question changes the rigour as well as the complexity of the options: What is 17% of 200? is a less rigorous question than What is 79% of 158? This goes for English, too:
What do the author Charles Dickens and his character Oliver Twist have in common?
a). Both were born in a workhouse
b). Both were separated from their parents and family
c). Both were put in prison for debt
d). Both had families who were put in prison for debt
e). Both were orphans
To answer that, you need a clear understanding of plot, character, author and context, and you have to think hard about them.
2. They make marking far less labour-intensive
For a start, they can be digitized and marked automatically by computer. You can give a 50 question quiz to 30 pupils and the 1500 answers are marked instantaneously. This is not true of extended, open questions at all: marking 30 paragraphs takes not 30 seconds but 30-60 minutes; marking 30 essays takes 120+. What’s more, instead of every teacher in the department, and every new teacher, slogging away marking the same assessment each year, the multiple choice quiz (for a novel like Oliver Twist or play like The Crucible – and many other topics in English, Maths and other subjects) can be created with an upfront workload, shared between a few subject experts, then used for years to come. Essay questions will still be important. But they can be combined with multiple-choice questions. Once created, MCQs save huge amounts of teacher time downstream.
3. They make pupil understanding more visible to teachers
The granularity of those 50 questions gives precise visibility into who understands what, and which pupils lack understanding. It’s a very precise diagnosis. If Jay gets 20 out of 50, it’s crystal clear that 40% isn’t as good as Izzy, who got 45 out of 50, and 90%. You can target support to Jay based on what he still hasn’t understood, instead of trying to reteach the lot. And if there were 5 questions that everyone got wrong, even Izzy, then you can reteach and revisit those in lessons. Multiple-choice questions are a formidably powerful diagnostic and formative tool for teaching.
[Update from Kevin Cooper’s comment below] Multiple choice questions also allow the full range of a course to be tested. An essay tests depth of understanding focused on a narrow selection of content; multiple choice questions test breadth of understanding across a much wider range of content. Both are important. But the middle option of open questions without options would ratchet up the labour-intensity for teachers, and not realise the benefits of multiple choice on reliability either.
All this challenges assumptions that are easy to make about such questions: that they aren’t rigorous or valid measures of learning. Rigour and validity depend on their design, which I’ll write about in another post: how to create effective multiple-choice questions.
Try using multiple-choice questions as an assessment vehicle. They are a useful part of the mix.
***
*But what if pupils guess the right answer? This can be mitigated in at least four ways, which can be combined:
- increase the number of questions (not just 1 or 2 but 10 or 20)
- increase the number of options (not just 4, a 25% chance of correct guessing, but 5, a 20% chance, or 6, a 16% chance of guessing)
- have a high pass mark: increase it from 70% to 80% or 90%
- have a penalty for incorrect answers: lose a mark for every wrong answer
http://yorkscience.org.uk have been using misconceptions built from exam answers to design multiple choice questions that test knowledge and how to explain that knowledge. This seems a better pay off in terms of teacher time – take a while to construct the multiple choice questions but they should have longevity and bear fruit in terms of easy to mark assessment.
Reblogged this on The Echo Chamber.
I really like the idea of using multiple choice in my teaching for the reasons you outline. One reason I don’t is because it takes lots of time to write good ones. However, I also need and want my students to actively recall without prompts. I need that because that is what they will need to do in their current exams. I want to test recall without prompts because it requires a greater degree of precise knowledge. In the same way that writing a sustained argument is harder than deciding if something is correct.
However, I am persuaded by your post and Daisy’s that multiple choice can play a really useful role. I just want someone else with more time to write them for me!
Thanks for summing this up so well – we’re just starting a lot of assessment overhauling in our online school and I’ll be sharing this one with everyone who’s going to be involved in the process.
Have used m/c formats with my geography classes for a number of years – and, as you say, Joe, they are a useful part of the mix. At KS3 each end of unit assessment is an essay to develop extended writing (and thinking) but it’s clear that some students – of the full ability range – don’t perform as well as their responses in the previous 5-7 weeks would have suggested. So we use a m/c quiz prior to the extended essay for a double purpose: it enables those who ‘know & understand’ a province in which to demonstrate this even if they are reluctant essay writers. I often find this is welcomed by some students who are particularly good at maths/science and like their understanding in short-shrift binary – it’s right/it’s wrong – and find it more daunting to go through the rigmarole of justifying arguments in lengthy prose. (Not saying this is right – but it’s a way of acknowledging their ability in the subject whilst developing their extended writing over the longer term). Secondly, the act of preparing for, doing, and going over a m/c quiz helps refresh information in students’ minds – and they then produce a better extended essay in the follow-up lessons.
In Y10&11 we’ve used m/c as revision tools for the last 3 years with students telling us that it’s one of the best revision techniques for lessons in the weeks prior to mock or actual exams. I’ve seen particular benefit when they do the quiz in pairs or 3s – and are to be heard arguing about which answer is correct and having to justify their choice to each other.
A few things we’ve learned along the way:
a) increase the challenge by stating there are 2 or more ‘correct’ answers out of the choice of 5 – but you’re not saying how many – and there are bonus points if they hit the exact number (ie no marks for selecting all of them)
b) for revision purposes it’s better to frame the Q&A around the negative: Q: Which of the following is NOT an effect of tropical storms…. so that students are actively thinking about 4 correct responses & bulking up on them, rather than singular correct points – helps particularly when going through the answers.
c) be careful to pilot the quiz on others – we sometimes ‘read’ questions one way, students read them another.
d) be prepared for the student who thinks laterally. Many a time I’ve had a hand raised to question an answer I’ve just declared ‘wrong’ to be given an alternative series of links that makes the answer plausibly correct ‘when you look at it that way’.
Useful line of development here as long as – as you say – it’s part of the mix, and not a substitute for extended writing assessment.
Writing questions in the negative is generally held to be poor practice. Much better to have them select all the correct options. Alternatively, use a true/false format where they actively evaluate each option.
You do need to take care when increasing the number of options that the challenge remains in the knowledge you want to test rather than dealing with a long array of options.
I agree with you and the others that commented. I think also that teachers need more training on writing good multiple choice tests, as you said it’s difficult. Also, I like the idea of scaffolding, such that a variety of assessments are given along the way. Perhaps a quiz early on is done in a multiple choice format, but the next assessment requires an oral recall response, and the third requires writing a thoughtful response. In my experience multiple choice is overused and the questions are often faulty, either too easy or as Andy Day mentioned vulnerable to lateral thinking. Too many teachers don’t test their tests.
Pingback: Why use multiple-choice questions? « I'll stick to 140 characters....
Great article. Glad that you finished with noting that multiple choice is only part of the mix. I teach year three and have found that m/c questioning can be useful orally as well as in written form. Particularly useful for chn who lack confidence when asked a question during a whole class discussion.
Me: “Patrick, what is the capital of France?”
Patrick: “Patrick looks like a rabbit in the headlights.”
Me: “Ok, how many choices do you want?”
Patrick “Three please.”
Me: “I think you can manage four. Is it Barcelona, Paris, Athens, or Rome?”
Patrick: “Paris.”
Me: “Amazing! I knew that you had that one. Well done.”
Great esteem booster, and shows me that he had it buried somewhere but couldn’t instantly recall. If I’d have gone to another child, for all I know Patrick thought the answer could have been Timbuktu.
As I’ve used it more, chn have begun to use this ‘narrow it down’ skill without my help. Meta-cognition is modelled, think about all the cities that you know, then eliminate the ones that you know it isn’t (e.g London) this should present you with a more narrow choice. Now, which of the three of four options is it?
However, coming from a philosophy background, I think that there is plenty of room for more open ended questioning, but I love the idea of m/c questioning where all of the options are correct.
E.g. Is killing wrong?
a) Yes, always.
b) Yes, but not animals.
c) No, it’s ok in war.
d) No, it’s ok in self defence.
e) …
And on and on. In fact, might plan that lesson now. Cheers!
I use multiple answer tests, and I just grade it like a normal test–no “bonus points”. It took me a while to figure out how to grade it, then I realized that each separate answer is a true/false question.
http://educationrealist.wordpress.com/2014/01/17/multiple-answer-math-tests/
Pingback: Why use multiple-choice questions? | Pragmatic Education | Learning Curve
I do follow the arguments you’ve made (and most of the comments here stating that we, teachers need more training). Still I have 2 important elements to add:
– Do check this research: https://mathed.asu.edu/media/pdf/pubs/clark/ClarkLinn2003.pdf and notice how the MC’s can obscure learning
– At the University in Ghent were I live they are banning all kinds of penalty for wrong answers as research has shown it limits the chances for the brighter students who are less confident in nature.
To add another benefit – MC allows the full RANGE of a course to be tested – in my subject (Geography) students can be very frustrated as only a fraction of the specification can actually be tested in the GCSE – even worse if some of that fraction happens to be the bit the student knows least well / was ill for / etc.
A great use of multiple choice questions for maths: http://www.diagnosticquestions.com/
Pingback: Multiple Choice Questions: A) Use regularly B) Don't use | HuntingEnglishHuntingEnglish
Pingback: How to evaluate a knowledge unit | Pragmatic Education
Pingback: ICT Training :: Friday 21st March – try
Pingback: How to design multiple-choice questions | Pragmatic Education
Pingback: QuickKey, multiple choice and hinge questions. | mrbenney
Pingback: A guide to this blog | Pragmatic Education
Pingback: A guide to this blog | Pragmatic Education
Pingback: Pedagogy Postcard #18: Problem Solving: Playing Detective | headguruteacher
Pingback: Increasing bandwidth – Planning a revision session. | @mrocallaghan_edu
Pingback: Teaching via the Trivium 21c | #hackthecurriculum
Pingback: Geography assessment tool | #hackthecurriculum
Pingback: Standin’ in line marking time…. | Distant Ramblings on the Horizon
Pingback: Assessment | PGCEPhysicalEducation
Pingback: Hornets and Butterflies: How to reduce workload | Pragmatic Education
Pingback: Using iPads for AFL – multiple choice questions | gavin holden
this is just a ridiculous article that is skewed and does not represent the better answer out of 2 or 3 answers, questions that most students get. Most multiple choice questions have more than one correct answer and the student is now deciding which one the test writer considers the most correct. They are for all intensive purposes trick questions that force a student to have a test taking strategy, to actually learn how to take a test. Pure laziness on the teachers part, they are plain and simple easier to grade.
Does anyone have examples of multiple choice for English Lang & Lit?
All I can find is Maths, History & Geography.
Pingback: 26 Strategies for meaningful manageable assessment by @powley_r – UKEdChat
Pingback: Golden needles in a haystack: Assessment CPD trove #4 | Joe Kirby