The Latest Change in the MBE

September 5th, 2016 / By Deborah J. Merritt

In the memo announcing results from the July 2016 MBE, Erica Moeser also notified law school deans about an upcoming change in the test. For many years the 200-question exam has included 190 scored items and 10 pre-test questions. Starting in February 2017, the numbers will shift to 175 scored items and 25 pre-test ones.

Pre-testing is an important feature of standardized exams. The administrator uses pre-test answers to gauge a question’s clarity, difficulty, and usefulness for future exams. When examinees answer those questions, they improve the design of future tests.

From the test-taker’s perspective, these pre-test questions are indistinguishable from scored ones. Like other test-makers, NCBE scatters its pre-test questions throughout the exam. Examinees answer each question without knowing whether it is a “real” item that will contribute to their score or a pre-test one that will not.

So what are the implications of NCBE’s increase in the number of pre-test items? The shift is relatively large, from 10 questions (5% of the exam) to 25 (12.5% of the exam). I have three concerns about this change: fair treatment of human research subjects, reliability of the exam, and the possible impact on bar passage rates. I’ll explore the first of these concerns here and turn to the others in subsequent posts.

Human Research Subjects

When examinees answer pre-test questions, they serve as research subjects. They answer those questions, not for their own benefit, but for the benefit of the exam administrator and future exam-takers. The examinees, moreover, have no power to refuse participation in the experiment; they can’t opt to take a shorter version of the exam with only scored items. If they want to be lawyers, they have to participate in the experiment.

Compelled participation in this experiment is particularly worrisome because (a) the overall experience is unpleasant, and (b) the test-taker pays for the privilege of participation. Think back to (or imagine) the days when dentists drilled teeth without novacaine and we paid for all of our dental work out of pocket. Would we have allowed dentists to drill all patients for an extra 30 seconds, not to benefit the patient but to test out the drill in ways that might help future patients?

The answer to that question is clearly no. Medical researchers follow strict guidelines when testing human subjects. So do social science researchers in universities. The rules are much looser when it comes to private testing agencies, but we should always think about the costs imposed on research subjects who lack the power to withhold consent.

What’s the Burden?

Historically, NCBE has required MBE-takers to devote 18 minutes of their time (5% of the six-hour exam) to aid future examinees. Going forward, that contribution will increase to 45 minutes (12.5% of the six-hour exam). That’s a substantial amount of time, especially when “volunteered” in the midst of a stressful, tiring experience.

If 175 questions are adequate to test examinees’ knowledge of MBE subjects, as NCBE now suggests, then why not let the test-takers go home after 5 hours and 15 minutes? Or let them have the additional 45 minutes to analyze the questions that count? Or even add the traditional 10 pre-test items to 175 real ones?

Pre-test questions add more than just time to an exam; they also add stress. These new questions may be more ambiguous or difficult than well-tested ones; the agency pre-tests them precisely to identify those defects. Exam-takers, however, can’t skip over challenging pre-test questions and focus on the “real” questions; they don’t know which are which. Answering flawed pre-test items can absorb disproportionate amounts of time and raise stress levels.

What Are the Benefits?

If pre-test questions simply tortured examinees, we would ban them. But it’s not that simple: these questions also confer benefits. For a standardized exam, there is no better way to test the suitability of questions. The most plausible alternative (administering exams to paid volunteers) is less reliable and prohibitively expensive.

Each group of examinees, moreover, benefits from the time that previous examinees devoted to pre-test items. The burdens and benefits of pre-testing thus are shared among the same population over time.

Note that this type of reasoning isn’t enough to justify nonconsensual medical research. We don’t let dentists drill their captive patients for the benefit of future patients–even if the patient sitting in the chair benefited from the dentist’s previous experiments. We wouldn’t allow that even if the benefits to future patients were high and there was no other effective way to conduct the research.

Standardized test makers get more leeway, but the balance of benefits and burdens still matters. The bar exam is a high-stakes, grueling experience. We shouldn’t take lightly any increase in the burdens imposed on test-takers–even if those costs produce benefits for future examinees.

The Balance

NCBE’s previous balance imposed a modest burden on exam-takers: just 10 questions and 18 minutes of time. The new burden is 2.5 times greater: 25 questions and 45 minutes of time.

The extent of the new burden isn’t unprecedented; LSAC requires LSAT-takers to devote 35 minutes to pre-test items. The MBE, however, is longer and more tiring than the LSAT. It also adjoins at least one more full day of testing. The stakes, finally, are even higher than those on the LSAT. Disappointed LSAT-takers can retake the test within 2-4 months; their lives, meanwhile, suffer little disruption. Examinees who fail the bar, in contrast, may lose their jobs. They also will have to wait a full six months to retake the exam.

I can’t judge NCBE’s new balance without knowing more about both the benefits it hopes to derive from additional pre-test items and the available alternatives (such as producing more forms of the test with different pre-test items on each one). Based on the information available to me, however, the new burden seems too high for nonconsensual subjects enduring a stressful, high-stakes exam.

Even more worrisome is NCBE’s failure to acknowledge this burden. Its memo to law school deans blithely announces that “[t]he change was made in consultation with our testing and measurement staff with the goal of further strengthening of the MBE.” That may be true, but did the staff weigh the impact on test-takers? Did they ask those test-takers for input?

We should never be cavalier about the testing of human subjects–especially when those subjects have no opportunity to withhold consent. The subjects who pre-test NCBE’s questions sit for six hours in uncomfortable testing centers, facing enormous time pressure to answer 200 questions that will determine their future. They spend another day writing essays in similarly uncomfortable–and time-pressured–conditions. They often drive long distances to perform these feats, or sleep away from home while taking the tests.

NCBE is adept at scoring, scaling, and equating numbers. The rest of us are just as preoccupied with cut scores and passing rates. There is a very human dimension, however, to all of those numbers. That’s a fact to remember whenever discussing the bar exam–but it’s especially important when we use examinees as research subjects.