Bar Exam Scores and Lawyer Discipline

June 3rd, 2017 / By Deborah J. Merritt

Robert Anderson and Derek Muller have posted a provocative paper, The High Cost of Lowering the Bar, in which they argue that “bar exam score is significantly related to likelihood of State Bar discipline throughout a lawyer’s career.” Thus, they warn, “lowering the bar examination passing score,” as several California law deans have advocated, “will likely increase the amount of malpractice, misconduct, and discipline among California lawyers.”

Anderson and Muller urge the state bar to collect more detailed data on the relationship between bar scores and lawyer discipline–and then to consider the possible impact on attorney misconduct if the Supreme Court lowers the passing score. “The data we have collected,” they conclude, “should raise serious concerns about the effect on consumers of lowering the passing score.”

What type of correlation did Anderson and Muller identify? Should it affect decisions about the passing score for the bar exam? Let’s take a closer look.

The Findings

Despite some suggestive language in the paper, Anderson and Muller do not identify a direct correlation between bar exam scores and disciplinary actions. As they concede, they lack the data to make that connection explicitly. Instead, they document two correlations: (1) a correlation between JD school and disciplinary actions; and (2) a correlation between exam sitting (February v. July) and discipline.

On the first point, Anderson and Muller find that graduates of schools that enroll students with low LSAT scores are more likely to receive discipline than those who graduate from schools that enroll students with higher LSAT scores. On the second point, they find that lawyers who passed the bar exam in February are more likely to elicit discipline than those who passed in July.

Neither of these relationships is particularly surprising to scholars who study professional responsibility. In 2013, Leslie Levin, Christine Zozula, and Peter Siegelman published a sophisticated study showing that attorneys who graduate from lower ranked law schools (as ranked by U.S. News) are significantly more likely to be disciplined than those graduating from more prestigious schools. LSAT scores correlate strongly with a school’s U.S. News rank, so Anderson and Muller’s first finding essentially replicates Levin’s study–although with fewer controls than Levin and her colleagues used.

Similarly, graduates of lower-ranked law schools are more likely than students from higher-ranked schools to take the February bar exam. (In Ohio, this correlation was .830 (p = .006) for the last year.) This relationship exists both because those lower-ranked schools have more repeaters and because the lower-ranked schools educate more part-time students who graduate in December. A correlation between February exam-takers and attorney discipline, once again, demonstrates a relationship between law school prestige and the likelihood of a graduate’s eventual discipline.

Causation?

What, then, does this correlation mean? It’s tempting to speculate that the bar exam measures some quality that directly affects attorney discipline. In particular, one might suggest that graduates of lower-ranked law schools know less legal doctrine than graduates of more elite institutions–or that they have poorer essay-writing skills. Those defects might eventually produce disciplinary charges.

Two key facts, however, weigh strongly against drawing that type of causal inference. First, as Anderson and Muller point out, “[t]here is virtually no discipline in the first 10 years of practice.” If the bar exam measured qualities related to attorney discipline, one would expect to see disciplinary cases emerge during those 10 years. Wouldn’t attorneys with marginal competency (as measured by the current bar exam) reveal their deficiencies during their early practice years?

Second, attorney discipline almost never rests on lack of knowledge about legal doctrine, poor reasoning skills, or bad writing–the skills currently measured by the bar exam. Levin and her colleagues reported that attorneys most often received discipline for failing to communicate with clients (20.0%), lack of diligence (17.93%), and failure to safeguard client property (11.26%). Only 4.14% of disciplinary sanctions related to “competence”–and even some of those cases may have reflected incompetence in areas that are not tested by the bar exam.

Details of recent disciplinary cases in California tell a similar tale. Attorneys are disciplined for misappropriating client funds, misrepresenting their CLE credits, defrauding business partners, and other sordid behavior. The reports are titillating: Frank Wilson, for example, is an attorney who assaulted two Farsi-speaking strangers in a parking lot and threatened to “beat the fucking shit out of [them].” But whatever the facts, the misconduct rarely relates to skills or knowledge tested on the current bar exam.

Correlation?

What, then, should we make of the correlation between law-school prestige and attorney discipline? As Anderson and Muller concede, there are many possible explanations. The most likely one stems from another well known fact among scholars of professional responsibility: Lawyers in solo practice and small firms receive over 90% of disciplinary sanctions. Since graduates of low-ranking law schools are more likely than their more elite peers to pursue those practices, they are also more likely to encounter discipline.

The link between solo/small practice and discipline is multi-faceted; Anderson and Muller mention some of the contributing factors (e.g., cash flow problems in solo practices and small firms; greater personal control over client funds in those firms; and socioeconomic discrimination by disciplinary authorities). Once again, however, none of these factors relate to performance on the bar exam. Solo practitioners do not elicit discipline because they write poor essays or lack knowledge of MBE subjects.

Is Correlation Enough?

Still, as Anderson and Muller point out, there is a correlation between the prestige of a lawyer’s JD school and the likelihood that s/he will be disciplined. Should that factor affect the cut score on the bar exam? I.e., should California maintain a relatively high cut score (or even raise the score) to keep out more graduates of lower-ranked schools–and thus keep predicted discipline rates low? Should the bar examiners consider likely discipline rates when setting the cut score?

The right answer to both questions is “no”–for three reasons. First, we should not predict malfeasance based on factors unrelated to that behavior. If bar applicants do not know the rules governing client funds, we should deny them admission until they learn those rules. But we should not assume, based on past correlations, that someone who graduated from a lower-ranked law school is more likely to abuse client funds than someone with a more prestigious pedigree.

This answer holds even if the bar examiners, using a broader dataset, identify a direct relationship between bar exam scores and attorney misconduct. Once again, exam scores merely correlate with discipline; it is unlikely that they measure a quality that directly produces malfeasance. We should not exclude individuals from a profession based on qualities that merely correlate with misconduct.

To underscore that point, consider this: One of the strongest predictors of attorney discipline is the y chromosome.* Male attorneys are much more likely than female ones to be disciplined. If we want to use correlations to reduce instances of attorney discipline, it would be much more efficient to ban men from the profession, subject them to special character exams, or require them to achieve a higher bar exam score than women. Those actions, of course, would raise special issues of gender discrimination–but they illustrate the drawbacks of predicting malfeasance based on correlations.

Second, there are better ways to address attorney misconduct than through manipulation of the bar exam’s cut score. Bar associations have created mentoring programs for lawyers who practice alone or in small groups. Every state maintains a lawyering assistance program to help lawyers struggling with addiction, mental health issues, or burnout–all factors that contribute substantially to attorney misconduct. Programs like these address misconduct where and when it most often occurs: among lawyers who have been practicing for more than a decade and who work within small practices.

Finally, there are recognized scientific ways to set the cut score for a licensing exam. The first step is to make sure that the exam measures the competencies that new lawyers need; an invalid exam cannot generate a meaningful cut score. Next, examiners choose among recognized methods for identifying a score that reflects minimum competence on the validated exam. Finally, licensing authorities can consider policy issues, including how much and what types of risk they want to accept: Should they set the cut score high, to prevent type one errors (detecting minimum competence that does not exist)? Or should they set the score lower to avoid type two errors (failure to detect minimum competence)? Both types of error can negatively affect consumers.

In making that policy decision, however, licensing authorities should not consider correlations that bear no causal relationship to the qualities that the exam measures. A test designed to measure specific knowledge and skills should not be used to predict malfeasance.

_________________________________________________

* Edited on 6/5/2017 at 8:00 p.m. I originally wrote that gender was the strongest predictor of attorney discipline, but that assertion depends upon what other variables are included in a predictive equation. Rather than push the point, I’ll leave it that gender is one of the strongest predictors. That doesn’t mean maleness causes discipline, just as graduating from a low-ranked law school doesn’t cause discipline. As the text stresses, both of these are simply correlations.