Do THIS, Not THAT When Writing Multiple-Choice Questions

06/24/2019

We often have to make sure that people can perform as needed before we let them perform on the job. For example, we don’t want to find out that bank tellers cannot count out the correct amount when working with customers. We certainly don’t want to assess people’s ability to put out a fire at work by setting a fire and seeing what they can do.

Assessment often needs to be practical, but it must also give us the information we need. In situations like fires or hazardous waste spills, it’s common to assess a simulated performance instead of a real performance.

Research shows that multiple-choice questions can, when well-written, simulate performance, especially if we are asking about decisions and steps to take. Well-written multiple-choice questions have a lot of advantages as an assessment method, including:

  • Easier scoring (and less bias in scoring) than other assessment formats
  • The ability to assess more content in a shorter amount of time
  • The capacity to analyze how well a question performs (using item statistics in a testing tool or LMS) so we can fix problems

Notice that I was just discussing, “well-written multiple-choice questions.” But most aren’t well-written, according to research. Research shows that too many multiple-choice questions have a host of flaws. And those flaws damage assessments, confuse participants and cause a lot of other problems for people and organizations.

5 major multiple-choice question flaws are found below:

  1.  They are unclear or otherwise poorly written.
  2. They are too easy to guess.
  3. They test recall of content not use of knowledge.
  4. They don't measure what they intend to measure.
  5. They end up testing something else, rather than evaluating to see if the test taker knows the content.

Let’s look at some examples of multiple-choice questions with typical flaws.

Example 1

You are using your outdoor wood-burning fire pit at your home for the first time. You should not operate the fire pit: (Select the best answer)

A) With seasoned wood

B) Under an overhang

C) With non-combustible materials near the fire pit

D) Using a screen to protect from sparks

The correct answer is B. You should not operate your wood-burning fire pit under an overhang. Overhanging structures or trees can easily ignite from flying sparks. Answers A, C, and D are safety measures for using an outdoor fire pit, and since they are correct, people are likely to select them and get the question wrong.

Research shows that negatively worded questions are harder to understand and easier to mess up when answering. A recent review of high stakes nursing exams lists negatively worded questions as the second most common question writing flaw.

Do this: Instead of writing negatively worded multiple-choice questions, phrase them positively. So, instead of asking, "Which of these should you not do…," consider using this prompt,  "Which of these is an approved safety measure…"

Example 2

Here’s another example of a poorly written multiple-choice question. Can you tell what’s wrong with it?

If sparks from the wood-burning fire pit ignite materials outside of the fire pit, you should: (Select the correct answer)

A) Unintended consequences of an outdoor fire

B) Any chairs should be moved away from the fire

C) Put out the fire with water from a garden hose or a fire blanket

The biggest problem with this question is that the wording gives away the answer, as only C satisfactorily completes the sentence. It’s obviously the correct answer as a result. And here’s a related problem: because two of the three choices do not complete the stem (the question being asked or problem to be solved), the three answer choices dwindle down to one. Not okay.

Do this: Make sure that wording does not give away the answer to the question.

Example 1 is unclear. Example 2 is too easy to guess. These problems make your assessments less valid. And invalid assessments are giant problems waiting to happen.

Validity is one of the most important, if not the most important, issues when writing multiple-choice questions. Validity means that the assessment measures what it says it measures. Valid assessments can measure deeper levels of learning and create deeper processing, which improves the ability to apply.

By Patti Shank

Source