Reflections of a Bar Exam Skeptic

May 26th, 2017 / By Deborah J. Merritt

Robert Anderson has posted a thoughtful comment on the bar exam in which he dubs me a “bar exam skeptic.” I accept the label with pride: I have been deeply skeptical of the bar exam for years. I first wrote about the exam in 2001, when the national pass rate for first-time takers was a relatively high 77% (see p. 23). My skepticism today, with a national pass rate of 69%, is no greater or smaller. As I wrote recently, it’s time to convene a National Task Force to examine our bar admissions process.

Who Cares About the Bar Exam?

As Professor Anderson rightly observes, decanal concerns about the bar exam have risen as pass rates have fallen. That’s human nature. The content and scoring of the bar exam are boring subjects for alumni gatherings, graduation speeches, or law review submissions. Few legal educators spontaneously write about setting cut scores, scaling essay questions, or equating test scores over time. The bar exam is like plumbing: most people take it for granted until something goes wrong.

But now the bar exam pipes are leaking and people are paying attention. The leak doesn’t mean we should patch things up just to revive pass rates; the bar exam should measure competence, not admit a predetermined number of lawyers. But now that people are paying attention, this is a good time to consider whether we’re using the right type of filter and piping in our rather antiquated system.

What Does Our Money Buy?

The current bar exam is surprisingly expensive. States charge applicants $300-$800 to take the exam, plus $100-$150 for the privilege of using a laptop. Many test-takers incur travel and lodging expenses because their states offer the exam in a single city. Bar review courses, which most applicants find essential to their success, add between $1400 and $3895 to the bill–depending on the state and provider.

But the biggest expense stems from lost earnings. Most applicants study full-time during the ten weeks preceding the exam, precluding other employment. At Ohio State, which admits students with “minimal” risk of bar failure (according to their LSAT scores), our May graduates are already holed up in study carrels listening to tapes, highlighting study materials, and creating flashcards. They are also flocking to evening lectures by our own “Bar Exam Wizard,” Katherine Silver Kelly; they attend these lectures on top of full-time bar review courses.

At the median annual salary of $65,000 for new law graduates, those ten weeks of bar study cost an applicant $12,500 in foregone income. Many will forego additional income as they wait for states to grade their exams and release the results. Some public agencies and small law firms, for example, continue to pay graduates as “student clerks” until they pass the bar. Other graduates don’t find any law-related work until they receive their passing notice.

The bar exam also imposes costs on law schools. Most schools now support at least one faculty or staff member who offers bar prep courses, review workshops, and/or individualized exam counseling. Some schools support two of those resource people.

I don’t fault graduates or schools for investing their time and money in bar study. Nor do I expect states to design and grade a bar exam for free. Any professional licensing system will cost money. But we should pause to consider just how much our current system costs–and to ask whether we are getting sufficient value for that money.

Just over 50,000 people took the bar exam for the first time last year. Using a conservative estimate of $15,000 in costs per taker, that’s a total of $750 million. If we include repeat takers (who numbered 24,000 last year and usually repeat their first-time expenses), the total jumps to $1.1 billion. What are we getting for that money?

Memorization

As Professor Anderson suggests, the bar exam mimics law school exams. That’s no accident: When states first designed bar exams, law schools offered the only model for how to test lawyering competence. The bar exam continues to focus on core subjects taught in law schools, as well as the type of essay and multiple choice questions that professors ask.

But there is a catch. Despite the congruence between law school exams and the bar exam, few law school graduates feel prepared to take the bar exam right after law school. That’s why they devote ten weeks to intensive study and shell out thousands of dollars for bar review courses.

How could this be? If the bar exam simply tests what is learned in law school, why aren’t graduates ready to take the exam a week after graduation–saving themselves thousands of dollars and putting their hard-earned knowledge to work more quickly? And why are law schools creating special courses or workshops to prepare students for the bar exam, if that preparation is already happening throughout the law school curriculum?

The answer lies in the M word, memorization. The bar exam requires applicants to store an astounding number of detailed legal principles in memory. The adult human brain is not good at memorization; most faculty members can’t remember the names of sixty students without several weeks of practice or the use of a seating chart.

Compared to their professors, law graduates are relatively adept at memorization. Many of them have honed those skills through closed-book exams in law school. Even the most adroit memorizers, though, don’t retain memorized material for long. A law school graduate who hasn’t thought about Property Law since her first year won’t remember many details of what she studied. She’ll find it easier to memorize principles the second time around than the first, because she already has a mental framework for those principles, but she still will have to work hard to embed those principles in memory.

As she prepares for the bar exam, that student will have to memorize the law of 10-12 different subjects at once. The bar exam no longer tests just basic principles in each subject; it requires test-takers to recall complex details. The need to memorize all of that material is why the bar exam is so arduous and expensive.

The bar exam, of course, isn’t just a memory game. Test-takers must apply those rules to hypothetical facts and, on the essay portions of the exam, explain their reasoning. The examinees, however, can’t perform those tasks unless they accurately recall the relevant rules of law.

Is this exercise, which relies so heavily on memorization, a valid test of an aspiring lawyer’s competence? Can’t we do better for a billion dollars a year? Or do better at a much lower cost?

Solutions

As I suggested in a recent essay, there are many ways to reduce the bar exam’s overreliance on memorization. We could develop multiple choice questions that test basic concepts rather than detailed rules. We could allow test-takers to refer to notes, codes, or other resources during the exam. We could divide the exam into smaller, subject-specific components and allow applicants to take those exam-lets during or after the relevant law school course–much as students take the MPRE while taking a course in Professional Responsibility. We could also expand the use of MPT exercises on the bar exam. Any of these approaches are likely to enhance the bar’s validity, although validity is a quality that needs to be proven rather than assumed.

We could also modify the exam to measure cognitive skills (like interviewing, fact gathering, counseling, and negotiating) that the current exam ignores. Law schools now teach those skills, students fill those classes, and employers increasingly seek them in hiring. There are creative ways to make sure that new lawyers possess those competencies–for far less than a billion dollars a year. Detailing those approaches, however, will require another column. Stay posted.