Experimental Design
Errors are incorrect answers to a multiple choice question. When designed well, they do more than make the question difficult: they help reveal how students think.
Description of the bug
In a multiple choice question, a a bug it is an incorrect answer written to appear plausible to students who have some misconception, partial understanding, procedural error, or reasoning gap.
A useful bug is not just a wrong answer. It is not good in a way that gives the teacher information. When students choose a diversion, the choice should suggest something meaningful about what they understand, don’t understand, or need to revisit.
Why Bugs Matter
Dynamic puzzles make multiple choice questions diagnostic. They can indicate whether the student misunderstood a concept, confused words, used an incorrect procedure, overruled, relied on extra recognition, or answered a different type of question than the one asked.
Common Types of Bugs
The best distractions tend to emerge from predictable patterns in the student’s thinking.
| Bug Type | What It May Reveal | For example |
|---|---|---|
| A Common Misconception | The student has a misunderstanding about the concept. | The student confuses area with perimeter. |
| Partial Understanding | The student understands part of the concept but misses the context or distinction. | The student chooses the correct answer in one situation but not in the described situation. |
| Word Confusion | The student does not read well or does not understand the academic term. | The student confuses “infer” with “summarize.” |
| Process Error | The student uses an incorrect method, step, formula, or sequence. | The reader repeats when the division is needed. |
| An Unsupported Explanation | The reader skips the available evidence. | The reader assumes the character’s motivation without the support of the text. |
Weak vs. Fixed Bugs
| A weak bug | Powerful Diverter |
|---|---|
| It’s obviously wrong or stupid | It is obvious to the reader who has some misunderstanding |
| Not related to the learning objective | It is related to the concept, skill, or reasoning that is being tested |
| It is not grammatically compatible with the title of the question | It has to do with grammar, length, tone, and structure |
| It works as a trick answer | It reveals something about the student’s thinking |
| It is easy to finish without understanding | It takes understanding to say no |
Example: Multiple Choice Item with Bug Analysis
In this example, each incorrect answer indicates a different type of learning problem.
Question: A student reads a short passage about a character who refuses help from a friend, even though the character is struggling. Which hypothesis is best supported by these data?
- The character doesn’t understand the problem.
- The actor values independence and does not want to appear weak.
- A friend tries to make the situation worse.
- The character will solve the problem at the end of the story.
Correct Answer: B. This answer is best supported because the reader must connect the character’s action—refusing help while struggling—to a logical explanation of the motivation. The answer requires reflection, not simple memorization.
| An option | Role | Almost Student Thinking | What It Reveals |
|---|---|---|---|
| A | A bug | The reader can see that the character is struggling but does not connect with the refusal to be helped and the motivation. | Part understanding |
| B | Correct Answer | The reader connects the character’s action with the logical explanation of the character’s thinking or motivation. | Definition based |
| C | A bug | The student assumes an argument or bad intention without sufficient evidence. | Unsupported reasoning |
| D | A bug | The reader predicts what may happen later rather than answering what current data supports. | Question type confusion |
Teaching Point: This item is useful because each incorrect option points to a different learning issue. Another student may see the problem but fail to articulate the motivation. One may think beyond the evidence. Others may confuse assumptions and predictions.
Checklist: How to Write Better Bugs
- Start with the intention of learning. Each puzzle must relate to the concept, skill, or thought process being tested.
- Use real student errors. Strong distractions often come from exit tickets, class discussion, drafts, quizzes, and common misconceptions.
- Keep the selection consistent. Answer choices should be consistent in grammar, length, tone, and specifics.
- Avoid tricks. The goal is to express the student’s thinking, not to punish students with small cubes of words.
- Review response patterns. Bugs are very useful when teachers see which wrong answers students have chosen and why.
Limitations
Distractions are only useful if they are meaningful and tied to the learning objective. Poorly worded errors may measure reading difficulty, test strategy, or attention to words rather than actual comprehension. The most useful bugs are reviewed after the test to identify patterns in students’ thinking.
Sources: Haladyna, TM, Downing, SM, & Rodriguez, MC (2002). Review of guidelines for multiple choice classroom assessments. Applied Measurement in Education, 15(3), 309–333. · Rodriguez, MC (2005). Three options are good for multiple choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3–13. · Brookhart, SM (2010). How to Assess Higher-order Thinking Skills in Your Classroom. ASCD.
