ExamSoft After All?
March 24th, 2015 / By Deborah J. Merritt
Why did so many people fail the July 2014 bar exam? Among graduates of ABA-accredited law schools who took the exam for the first time last summer, just 78% passed. A year earlier, in July 2013, 82% passed. What explains a four-point drop in a single year?
The ExamSoft debacle looked like an obvious culprit. Time wasted, increased anxiety, and loss of sleep could have affected the performance of some test takers. For those examinees, even a few points might have spelled the difference between success and failure.
Thoughtful analyses, however, pointed out that pass rates fell even in states that did not use ExamSoft. What, then, explains such a large performance drop across so many states? After looking closely at the way in which NCBE and states grade the bar exam, I’ve concluded that ExamSoft probably was the major culprit. Let me explain why–including the impact on test takers in states that didn’t use ExamSoft–by walking you through the process step by step. Here’s how it could have happened:
Tuesday, July 29, 2014
Bar exam takers in about forty states finished the essay portion of the exam and attempted to upload their answers through ExamSoft. But for some number of them, the essays wouldn’t upload. We don’t know the exact number of affected exam takers, but it seems to have been quite large. ExamSoft admitted to a “six-hour backlog” and at least sixteen states ultimately extended their submission deadlines.
Meanwhile, these exam takers were trying to upload their exams, calling customer service, and worrying about the issue (wouldn’t you, if failure to upload meant bar failure?) instead of eating dinner, reviewing their notes for the next day’s MBE, and getting to sleep.
Wednesday, July 30, 2014
Test takers in every state but Louisiana took the multiple choice MBE. In some states, no one had been affected by the upload problem. In others, lots of people were. They were tired, stressed, and had spent less time reviewing. Let’s suppose that, due to these issues, the ExamSoft victims performed somewhat less well than they would have performed under normal conditions. Instead of answering 129 questions correctly (a typical raw score for the July MBE), they answered just 125 questions correctly.
August 2014: Equating
The National Conference of Bar Examiners (NCBE) received all of the MBE answers and began to process them. The raw scores for ExamSoft victims were lower than those for typical July examinees, and those scores affected the mean for the entire pool. Most important, mean scores were lower for both the “control questions” and other questions. “Control questions” is my own shorthand for a key group of questions; these are questions that have appeared on previous bar exams, as well as the most current one. By analyzing scores for the control questions (both past and present) and new questions, NCBE can tell whether one group of exam takers is more or less able than an earlier group. For a more detailed explanation of the process, see this article.
These control questions serve an important function; they allow NCBE to “equate” exam difficulty over time. What if the Evidence questions one year are harder than those for the previous year? Pass rates would fall because of an unfairly hard exam, not because of any difference in the exam takers’ ability. By analyzing responses to the control questions (compared to previous years) and the new questions, NCBE can detect changes in exam difficulty and adjust raw scores to account for them.
Conversely, these analyses can confirm that lower scores on an exam stem from the examinees’ lower ability rather than any change in the exam difficulty. Weak performance on control questions will signal that the examinees are “less able” than previous groups of examinees.
But here’s the rub: NCBE can’t tell from this general analysis why a group of examinees is less able than an earlier group. Most of the time, we would assume that “less able” means less innately talented, less well prepared, or less motivated. But “less able” can also mean distracted, stressed, and tired because of a massive software crash the night before. Anything that affects performance of a large number of test takers, even if the individual impact is relatively small, will make the group appear “less able” in the equating process that NCBE performs.
That’s step one of my theory: struggling with ExamSoft made a large number of July 2014 examinees perform somewhat below their real ability level. Those lower scores, in turn, lowered the overall performance level of the group–especially when compared, through the control questions, to earlier groups of examinees. If thousands of examinees went out partying the night before the July 2014 MBE, no one would be surprised if the group as a whole produced a lower mean score. That’s what happened here–except that the examinees were frantically trying to upload essay questions rather than partying.
August 2014: Scaling
Once NCBE determines the ability level of a group of examinees, as well as the relative difficulty of the test, it adjusts the raw scores to account for these factors. The adjustment process is called “scaling” and it consists of adding points to the examinees’ raw scores. In a year with an easy test or “less able” examinees, the scaling process adds just a few points to each examinee’s raw score. Groups who faced a harder test or were “more able,” get more points. [Note that the process is a little more complicated that this; each examinee doesn’t get exactly the same point addition. The general process, however, works in this way–and affects the score of every single examinee. See this article for more.]
This is the point at which the ExamSoft crisis started to affect all examinees. NCBE doesn’t scale scores just for test takers who seem less able than others; it scales scores for the entire group. The mean scaled score for the July 2014 MBE was 141.5, almost three points lower than the mean scaled score in July 2013 (which was 144.3). This was also the lowest scaled score in ten years. See this report (p. 35) for a table reporting those scores.
It’s essential to remember that the scaling process affects every examinee in every state that uses the MBE. Test takers in states unaffected by ExamSoft got raw scores that reflected their ability, but they got a smaller scaling increment than they would have received without ExamSoft depressing outcomes in other states. The direct ExamSoft victims, of course, suffered a double whammy: they obtained a lower raw score than they might have otherwise achieved, plus a lower scaling boost to that score.
Fall 2014: Essay Scoring
After NCBE finished calculating and scaling MBE scores, the action moved to the states. States (except for Louisiana, which doesn’t use the MBE), incorporated the artificially depressed MBE scores into their bar score formulas. Remember that those MBE scores were lower for every exam taker than they would have been without the ExamSoft effect.
The damage, though, didn’t stop there. Many (perhaps most) states scale the raw scores from their essay exams to MBE scores. Here’s an article that explains the process in fairly simple terms, and I’ll attempt to sum it up here.
Scaling takes raw essay scores and arranges them on a skeleton provided by that state’s scaled MBE results. When the process is done, the mean essay score will be the same as the mean scaled MBE score for that state. The standard deviations for both will also be the same.
What does that mean in everyday English? It means that your state’s scaled MBE scores determine the grading curve for the essays. If test takers in your state bombed the MBE, they will all get lower scores on the essays as well. If they aced the MBE, they’ll get higher essay scores.
Note that this scaling process is a group-wide one, not an individual one. An individual who bombed the MBE won’t necessarily flunk the essays as well. Scaling uses indicators of group performance to adjust essay scores for the group as a whole. The exam taker who wrote the best set of essays in a state will still get the highest essay score in that state; her scaled score just won’t be as high as it would have been if her fellow test takers had done better on the MBE.
Scaling raw essay scores, like scaling the raw MBE scores, produces good results in most years. If one year’s graders have a snit and give everyone low scores on the essay part of the exam, the scaling process will say, “wait a minute, the MBE scores show that this group of test takers is just as good as last year’s. We need to pull up the essay scores to mirror performance on the MBE.” Conversely, if the graders are too generous (or the essay questions were too easy), the scaling process will say “Uh-oh. The MBE scores show us that this year’s group is no better than last year’s. We need to pull down your scores to keep them in line with what previous graders have done.”
The scaled MBE scores in July 2014 told the states: “Your test takers weren’t as good this year as last year. Pull down those essay scores.” Once again, this scaling process affected everyone who took the bar exam in a state that uses the MBE and scales essays to the MBE. I don’t know how many states are in the latter camp, but NCBE strongly encourages states to scale their essay scores.
Fall 2014: MPT Scoring
You guessed it. States also scale MPT scores to the MBE. Once again, MBE scores told them that this group of exam takers was “less able” than earlier groups so they should scale down MPT scores. That would have happened in every state that uses both the MBE and MPT, and scales the latter scores to the former.
Conclusion
So there you have it: this is how poor performance by ExamSoft victims could have depressed scores for exam takers nationwide. For every exam taker (except those in Louisiana) there was at least a single hit: a lower scaled MBE score. For many exam takers there were three hits: lower scaled MBE score, lower scaled essay score, and lower scaled MPT score. For some direct victims of the ExamSoft crisis, there was yet a fourth hit: a lower raw score on the MBE. But, as I hope I’ve shown here, those raw scores were also pebbles that set off much larger ripples in the pond of bar results. If you throw enough pebbles into a pond all at once, you trigger a pretty big wave.
Erica Moeser, the NCBE President, has defended the July 2014 exam results on the ground that test takers were “less able” than earlier groups of test takers. She’s correct in the limited sense that the national group of test takers performed less well, on average, on the MBE than the national group did in previous years. But, unless NCBE has done more sophisticated analyses of state-by-state raw scores, that doesn’t tell us why the exam takers performed less “ably.”
Law deans like Brooklyn’s Nick Allard are clearly right that we need a more thorough investigation of the July 2014 bar results. It’s too late to make whole the 2,300 or so test takers who may have unfairly failed the exam. They’ve already grappled with a profound sense of failure, lost jobs, studied for and taken the February exam, or given up on a career practicing law. There may, though, be some way to offer them redress–at least the knowledge that they were subject to an unfair process. We need to unravel the mystery of July 2014, both to make any possible amends and to protect law graduates in the future.
I plan to post some more thoughts on this, including some suggestions about how NCBE (or a neutral outsider) could further examine the July 2014 results. Meanwhile, please let me know if you have thoughts on my analysis. I’m not a bar exam insider, although I studied some of these issues once before. This is complicated stuff, and I welcome any comments or corrections.
Updated on September 21, 2015, to correct reported pass rates.
Needing Law Schools
March 22nd, 2015 / By Deborah J. Merritt
I agree entirely with Noah Feldman that society needs law schools. He couldn’t have said it better. This, however, is exactly why law schools need to fix their financial model. Most schools lack the big endowments of Harvard and other elite schools. Students, meanwhile, are increasingly unwilling to pay so much more tuition than Feldman did in the 1990’s or I did in the 1970’s. We need to keep asking: Why does it cost so much more today to learn what the law “can be”?
I learned a lot about what the law can be from Ruth Bader Ginsburg, my constitutional law professor at Columbia in 1979. I also learned from Herbert Wechsler, author of the much-cited article on “neutral principles” in constitutional law; William Carey, one of the New Deal architects and an early chair of the SEC; E. Allan Farnsworth, Reporter for the Restatement (Second) of Contracts; Maurice Rosenberg, one of the earliest legal scholars to apply social science research to legal problems; and many others. Why were all of these luminaries able to teach me and my classmates for so much less tuition than Columbia and other schools demand today?
In part, they earned less. I know that, because I am the daughter of yet another Columbia professor from that era: William K. (“Ken”) Jones. Our family did just fine financially, but we didn’t have the affluence that law professors enjoy today. Another explanation rests on the enormous number of staff members that law schools now need to operate. Communications staff, admissions staff, development staff, student services staff . . . . Each seems indispensable in the modern law school, but how many contribute to our mission of teaching students and others what the law can be?
I doubt that it’s possible to unwind the contemporary law school, to dismiss all of the staff, and go back to an earlier, simple world. Although it’s a charming notion, isn’t it? We could simply post our lower tuition, admit students who apply (without spending time marketing to them), teach them, and send them into the world knowing something about both what the law is and what it can be. Meanwhile, we would publish and engage in law reform efforts–as Ginsburg, Wechsler, Carey, and the others did–while teaching four courses a year.
I know that’s unlikely to happen, so we’ve got to find other ways to fix the financial model. Shifting the first year of law school to the undergraduate curriculum makes sense to me. Let’s teach more people about both the power of law and what it can be. Meanwhile, let’s lower tuition for those who will actually practice law. We, as professors, can teach people what the law can be–but our graduates are essential to make those changes happen.
How Many Lawyers?
March 20th, 2015 / By Deborah J. Merritt
A few years ago, I used employment projections from the Bureau of Labor Statistics to project the number of job openings for licensed lawyers during the current decade. At the time, BLS was the best available source for that type of projection; it remains a useful resource today. The BLS makes these predictions precisely to help workers, employers, and policymakers understand the likely demand for workers in particular occupations.
Why should law schools care about these predictions? As Michael Simkovic and Frank McIntyre show in two recent papers, a JD historically has conferred financial advantage (compared to entering the workforce with just a BA) even if graduates did not work as practicing lawyers. If law graduates reap financial returns from their degrees, regardless of the jobs they take, does it matter how many jobs they find as practicing lawyers?
Some scholars (including me) wonder whether financial returns to a law degree will remain as high as they have been. But let’s leave that debate aside for now. Let’s consider, instead, why law schools should care about the number of grads who will find jobs as licensed lawyers–and what that number might be.
Who Cares?
Law graduates, students, and prospective students seem to care. Several sources indicates that, although some graduates enjoy work that does not require bar admission, graduates overall prefer to practice law. Every year, for example, NALP publishes a table showing how many graduates are employed but still seeking work. These tables consistently show that grads in jobs that require bar admission are far less likely than other grads to be seeking new jobs. In 2010, for example, just 15.1% of the graduates in “bar required” work were seeking other employment, while 48.1% of those in “JD preferred” jobs and 49.0% of grads in “other professional positions” were doing so. [Sorry, source not available online]
Similar preferences emerge later in the career. The After the JD longitudinal study found that law grads working in non-lawyering jobs were less satisfied than those in lawyering positions. The distinction held both seven and twelve years after law school graduation. [No online link for that one either, I’m afraid, but check p. 70 of the AJD II and AJD III reports.]
If graduates care, I suspect that students and prospective students care as well. And if only out of self interest, law schools should care too. We can tell students that a law degree offers many options, and that historically those options have paid off financially compared to a BA alone, but the current crop of students isn’t buying that pitch. If they want to practice law, they’re still coming to law school. But if they’re looking for intellectual stimulation, a degree with diverse options, or other benefits, they seem to be going elsewhere.
This is partly why I think law schools should examine the way we package the education we offer. We’re good at teaching close reading, careful writing, and critical thinking–and we could teach those skills to undergraduates. Talented undergrads want those skills, and employers will pay a premium for them. At the same time, we’re good (and could be better) at teaching advanced legal doctrine and other intellectual skills essential for law practice. We can keep providing that education to a more focused group of JD students who primarily will become lawyers.
So How Many Lawyers?
This brings me back to the question I started with: If prospective law students care about whether they will get jobs as licensed attorneys, and if law schools should care about that question (if only to attract students), about how many law graduates are able to get jobs that require bar admission?
My recent study of law graduates in the State of Ohio gave me some numbers to work with. I tracked job outcomes for all 1,214 new lawyers who passed the Ohio bar exam in 2010. In December 2014, about 75% of them held jobs that required a law license. Most of the rest were employed, but in other types of work.
My population included only licensed lawyers, not law graduates who didn’t take or pass a bar exam. One thing we know about the latter group is that they can’t be practicing law. So, after performing some calculations to account for that group, I estimated that about two-thirds of 2010 graduates were practicing law four and a half years after graduation.
Will the same percentage hold for graduates from other years and in other states? I don’t know; we often have to deal with limited data and isolated points of reference in making real-world plans. I explain in the paper why I think Ohio offers a useful perspective, and why I think the annual number of new bar-required jobs will remain stable in coming years. I’ll write more about both of those issues here soon.
The Good News
The good news for both law schools and prospective students is that my estimate is higher than the BLS’s historic projections. I estimate that, four and a half years after graduation, about 29,250 members of the Class of 2010 are practicing law. Some of the jobs are dubious solo practices; some undoubtedly are part-time, temporary, and/or low paid. Some of them are jobs that a graduate secured only after failing and re-taking the bar exam. Prospective students need to take those factors into account, not simply consider whether they’ll be able to find a job practicing law.
When all is said and done, though, graduates are finding more “lawyering” jobs than BLS once predicted–although not as many as BLS predicts through a proposed revision to its forecasting method. I’ll comment on the latter in another post.
The other piece of good news is that, if my calculations are correct, and if law school attrition rates remain constant, then about 84.5% of the current 1L class will find lawyering jobs within a few years after graduation. That level of job placement may be sufficiently attractive to maintain enrollment at current rates.
The Bad News
I couched that last sentence carefully: “to maintain enrollment at current rates.” Better placement in lawyering jobs will reassure prospective law students, but I doubt it will draw them back to law school in droves. Law schools, meanwhile, will need to worry about bar passage rates as they enroll students with lower credentials. Declining bar passage rates will discourage potential applicants both directly and indirectly, as they depress the percentage of graduates working in jobs that require bar passage.
Schools will also vary in the percentage of graduates they place in jobs requiring bar admission. Some will place more than 85% in those positions; others will place much less. If I’m right that potential students care about getting jobs that require a law license, enrollment declines will continue at the latter schools.
Summing Up
In making predictions, both here and in my paper, I offer some very specific numbers. I do that to offer a point of reference for debate; I can’t say exactly how many members of the Class of 2017 will find lawyering jobs, or how many students will apply to law school in 2018. I do think, though, that JD students care about their odds of securing a job that requires a law license and that law schools need to account for that preference. To do that, it helps to know as much as we can about operation of the legal market.
Bankruptcy Petitions
March 17th, 2015 / By Deborah J. Merritt
Michael Sousa has written a thoughtful paper about the issue of non-lawyers preparing bankruptcy petitions for consumer debtors. As Sousa acknowledges, some of these preparers urge their clients to commit fraud; others serve their clients incompetently. But then again, the same is true of some bankruptcy lawyers–and the lawyers charge more than many consumers can afford.
After reviewing the field, Sousa proposes creation of a cadre of licensed petition preparers for no-asset Chapter 7 bankruptcy filings. The United States Trustee’s Office could regulate these practitioners, requiring them to demonstrate adequate knowledge of the law and adherence to appropriate ethical standards. The practitioners, however, would not need JD’s or law licenses.
Sousa’s proposal makes sense to me. I note it here because he has identified yet another area in which non-lawyers may be able to handle the law-related needs of clients who cannot afford an attorney’s fee. As Sousa details, the federal government already allows qualified non-lawyers to represent clients before the IRS, US Patent and Trademark Office, and (in immigration matters) the Department of Homeland Security. The transfer of law-related business to non-lawyers has been slow but steady.
What Happened to the Class of 2010?
March 14th, 2015 / By Deborah J. Merritt
It has been almost five years since the Class of 2010 walked across the stage to pick up their JDs. Since then, we’ve debated whether the weak job outcomes for that class stemmed solely from the recession or represented structural changes within the profession itself. We’ve also wondered whether the graduates would improve their employment status as they moved into the workforce and the economy picked up. I decided to find out.
1,214 New Lawyers
In a paper just posted to SSRN, I tracked job outcomes for all 1,214 new lawyers who passed the Ohio bar exam in 2010 and then joined the bar. The study doesn’t focus on the graduates of a particular school, as some analyses do. I included all bar-passers, regardless of the school they attended (although note that Ohio only allows graduates of ABA-accredited schools to take the exam). I also followed successful examinees even if they left Ohio.
My focus on bar admittees parallels the approach taken in several earlier studies of the legal profession, including the recent After the JD (AJD) surveys. As I note in the paper, that means I can’t say very much about the approximately 12% of law school graduates who never gain bar admission. It also means that I overstate the percentage of jobs that require bar admission, although I offer some corrections for this in the paper.
I also explain in the paper why Ohio offers a useful perspective on trends nationally. The state’s 9-month outcomes, as reported by NALP, are similar to national averages. Ohio is the ninth largest state for both number of practicing lawyers and number of jobs taken by new law graduates. It’s also home to the nation’s largest law firm and several other BigLaw shops. And the state has a healthy economy, with an overall unemployment rate (4.8% in December 2014) that is lower than in states with the largest legal markets. During the same month, New York’s unemployment rate was 5.8%; Illinois, 6.2%; California, 7.0%; and DC, 7.3%.
1,137 Jobs
For each of the 1,214 lawyers in my study, I searched for the job held in December 2014. I didn’t use surveys; I relied entirely on public online sources. This turned out to be much easier than I thought it would be–and produced a much more complete dataset than surveys do. With Ohio’s bar directory, employer websites, directories like LinkedIn, and other sources (e.g., alumni stories published online by law schools), I found a December 2014 job for 1,137 graduates or 93.7% of the population. I explain in the paper why the remaining 6.3% are most likely unemployed (for some, I found direct evidence of that) or serve as a proxy for the percentage unemployed.
Enough of the preliminaries. What did I find? First, it’s hard to read the job histories without feeling great sadness for the Class of 2010. Sure, many of them have ended up in decent jobs. Some may even hold their dream job. But they’ve had (and are still having) a very tough time. Almost a tenth bill themselves as “solo practitioners,” although I found no evidence of an active practice (such as a website) for half of them. At best, these solos are struggling to establish themselves at an early stage of their careers. At worst, they’re unemployed job seekers doing occasional legal work for friends.
Meanwhile, two-thirds of the graduates have switched jobs at least once in four years–that’s twice the rate of turnover that AJD reported for the Class of 2000 during their early careers. Among those who changed jobs, the average number of jobs was 2.7–almost three jobs in just four years. Before you shake your head over those peripatetic millennials, note that median job tenure for 25-34-year-olds nationally is 3.0 years; the millennials actually stick to their jobs somewhat longer than young adults did before 2010.
It’s easy to talk about the job market from the comfort of well paid, tenured positions. It’s a lot harder to be out there in the trenches. Before we talk more about numbers, let’s recognize the fortitude (and ongoing struggles) of the people in the Class of 2010 and other recent classes.
Structural Change
I will write a series of posts highlighting different parts of my findings, but here’s the bottom line: I found strong evidence of structural changes in the employment market. Here are the top indicators:
1. The Class of 2010 graduated almost five years ago, a year after the Great Recession officially ended. It has been a slow and jobless recovery, but there have been signs of growth for quite some time now. There are certainly clients hiring lawyers: the top-earning lawyers are doing very nicely, and a wide range of alternative-service providers are prospering. Yet recent grads are lagging in job outcomes when compared to earlier classes at a similar career point.
2. Only three-quarters of the licensed lawyers hold jobs that require a law license. After adjusting for graduates who never took or passed the bar, I estimate that no more than two-thirds of 2010 graduates work in jobs that require bar admission. We can argue about whether future students will be willing to invest in law school for early-career jobs that don’t use their full education and licensing (I don’t believe they will). It’s clear, however, that the percentage of recent law graduates practicing law has been declining for some time. Whether that makes you smile or weep, it’s a shift in the market.
3. Only 40.4% of the licensed graduates work in law firms, a barely perceptible increase over the 39.5% of the class who reported those jobs nine months after graduation. In fact, the 40.4% almost certainly masks a decrease in law firm employment, because I obtained data only on licensed lawyers; the 9-month figure includes all law school graduates. Even at 40.4%, this is a striking figure. Despite four years of experience and economic recovery, the Class of 2010 made no headway in securing law firm jobs.
4. The low level of law firm employment is even more remarkable when compared to outcomes for the Class of 2000, which was tracked by the AJD study. That class graduated into relatively good economic times, but weathered a recession and jobless recovery during their first years in the profession. When the class reported their three-year outcomes to AJD in late 2002 and 2003, national unemployment levels were actually higher than when I identified jobs held by the Class of 2010 in December 2014. Yet 62.1% of the Class of 2000 practiced law with a firm three years after graduation–compared to just 40.4% of the Class of 2010 four years out. That’s a phenomenal drop of almost twenty-two percentage points.
5. The shift in law firm employment was not due solely (or perhaps even primarily) to changes in BigLaw employment patterns. Ohio, like most states, is not home to a law school ranked among the US Not-News top twenty. Yet we have all of those tasty BigLaw offices and all types of other employers. After the crash, did elite law school graduates jump to Ohio, taking our best jobs and creating a market cascade? I once thought that might be the case, but the evidence says otherwise. The number of elite school graduates passing the Ohio bar actually peaked in 2007–and was low even then. The changes we are seeing in lawyer employment are systemic; they arise from shifts at many levels of the job market.
There’s More
Much more, but I’ll save those data for another day. Meanwhile, you’ll find the full paper here.
Scholarship: Cost and Value
March 12th, 2015 / By Deborah J. Merritt
Critics of legal education raise two key questions about our scholarship: (1) How much value does it offer? And, (2) do law schools have to spend so much money to produce that value?
The answer to the second question is easy: No. We used to produce plenty of superb scholarship with typewriters and four-course teaching loads. Now that we have laptops, tablets, high-powered statistical software, and 24/7 online libraries, our productivity has leaped. Law schools could easily restore teaching loads to four courses a year while still facilitating plenty of good research. The resulting reduction in faculty size could help fund scholarships and reduce tuition.
The answer to the value question is harder. Do we mean immediate pay-off or long term influence? Do we care about value to judges, legislators, practicing attorneys, clients, teachers, students, or some other group? Does each article have to demonstrate value? Or do we recognize that trial and error is part of scholarship as well as other endeavors?
Those are difficult questions and they deserve a series of posts. For now, I’ll limit my discussion to a recent paper by Jeffrey Harrison and Amy Mashburn, which has already provoked considerable commentary. I agree with some of Harrison and Mashburn’s observations, but the empirical part of their paper goes badly astray. Without better method, their conclusions can’t stand. In fact, as I note below, some of their findings seem at odds with their recommendations.
Measuring Citation Strength
Harrison and Mashburn decided to measure the strength of citations to scholarly work, rather than simply count the number of citations. That was an excellent idea; scholars in other fields have done this for decades. There’s a good review of that earlier work in Bornmann & Daniel, Do Citation Counts Measure? A Review of Studies on Citing Behavior, 64 Journal of Documentation 45 (2008). (By the way, isn’t that an amazing name for a journal?)
If Harrison and Mashburn had consulted this literature, they would have found some good guideposts for their own approach. Instead, the paper’s method will make any social scientist cringe. There’s a “control group” that is nothing of the sort, and the method used for choosing articles in that group is badly flawed.* There is little explanation of how they developed or applied their typology (written protocol? inter-rater agreement? training periods?). Harrison and Mashburn tell us only that the distinctions were “highly subjective,” the lines were “difficult to draw,” and “even a second analysis by the current researchers could result in a different count.” Ouch.
Is it possible to make qualitative decisions about citation strength in a thoughtful, documented way? Absolutely. Here’s an example of a recent study of citation types that articulates a rigorous method: Stefan Stremersch, et al., Unraveling Scientific Impact: Citation Types in Marketing Journals, 32 Int’l Journal of Research in Marketing 64 (2015). Harrison and Mashburn might choose a different design than previous scholars, but they need to develop their parameters, articulate them to others, and apply them in a controlled way.
Influence and Usefulness
Harrison and Mashburn conclude that most legal scholarship “is not regarded as useful.” Even when a judge or scholar cites an article, they find, most of the cited articles “serve no useful function in helping the citing author advance or articulate a new idea, theory or insight.” Application of this standard, however, leads to some troubling results.
The authors point, for example, to an article by John Blume, Ten Years of Payne: Victim Impact Evidence in Capital Cases, 88 Cornell L. Rev. 257 (2003). A court cited this article for the seemingly banal fact that “the federal government, the military, and thirty-three of the thirty-eight states with the death penalty have authorized the use of victim impact evidence in capital sentencing.” Harrison and Mashburn dismiss this citation as “solely to the descriptive elements of the article.”
That’s true in a way, but this particular “description” didn’t exist until Blume researched all of that state and federal law to create it. The court wanted to know the state of the law, and Blume provided the answer. This answer may not have “advance[d] . . . a new idea, theory or insight,” but most cases don’t require that level of theory. Disputes do require information about the existing state of the law and Blume assembled information that helped advance resolution of this dispute. Why isn’t that a worthwhile type of influence?
I suspect that judges and practitioners appreciate the type of survey that Blume provided; analyzing the law of 40 jurisdictions requires both time and professional judgment. Blume, of course, did more than just survey the law: he also pointed out crevices and problems in the existing law. But dismissing a citation to the survey portion of his article seems contrary to the authors’ desire to create scholarship that will be more useful.
A reworked method might well distinguish citations to descriptive/survey research from those that adopt a scholar’s new theory. Asking scholars to limit their work to the latter, however, seems counter productive. A lot of people need to know what the law is, not just what it might be.
Judges and Scholars
One statistic in the Harrison and Mashburn article blew me away. On page 25, they note that 73 out of 198 articles from their “top 100” group of journals were cited by courts. That’s more than a third (36.9%) of the articles! I find that a phenomenally high citation rate. I know from personal experience that judges do pay attention to law review articles. When I clerked for Justice O’Connor, for example, she asked us to give her a shelf of law review articles for each of the bench memos we wrote. She didn’t want just our summaries of the articles–she wanted the articles themselves.
But I never would have guessed that the judicial citation rate was as high as 36.9% for professional articles, even for journals from the top 100 schools. At least in judicial circles, there’s a big drop-off between learning from an article and citing the article. Most judges try to keep their opinions lean, and there’s no cultural pressure to cite scholarly works.
I’m not sure how to mesh the judicial citation statistic with the tone of Harrison and Mashburn’s article. More than a third sounds like a high citation rate to me–as does the one quarter figure for journals in the 15-100 group.
Ongoing Discussion
Harrison and Mashburn urge critical debate over the value and funding of legal scholarship, and I back them all the way on that. I wrote this post in that spirit. As I note above, I don’t think law schools need to spend as much money as they do to produce strong levels of excellent scholarship. I also applaud efforts to replace citation counting with more nuanced measures of scholarly value. But we need much stronger empirical work to examine claims like the ones advanced in this paper. Are Harrison and Mashburn right that most legal scholarship “is not regarded as useful”? I don’t know, but I was put off by strong statements with weak empirical evidence.
__________________________
* Harrison and Mashburn chose the first article from each volume. That’s a textbook example of non-random selection: the first article in a volume almost certainly differs, on average, from other articles.
Timing Law School
March 10th, 2015 / By Deborah J. Merritt
Michael Simkovic and Frank McIntyre have a new paper analyzing historic income data for law school graduates. In this article, a supplement to their earlier paper on the lifetime value of a law degree, Simkovic and McIntyre conclude that graduates reap most of the value of a JD whether they graduate in good economic times or poor ones. (Simkovic, by the way, just won an ALI Young Scholar Medal. Congratulations, Mike!)
Simkovic and McIntyre’s latest analyses, they hope, will reassure graduates who earned their degrees in recent years. If history repeats, then these JDs will reap as much financial benefit over their lifetimes as those in previous generations. Simkovic and McIntyre also warn prospective students against trying to “time” law school. It’s difficult to estimate business cycles several years in advance, when a 0L must decide whether to take the plunge. And, again according to historical data, timing won’t make much difference. Under most circumstances, delay will cost more financially than any reward that successful timing could confer.
But Is This Time Different?
History does repeat, at least in the sense of economic conditions that cycle from good to bad and back again. There’s no doubt that recent law school graduates have suffered poor job outcomes partly because of the Great Recession and slow recovery. It’s good to know that graduates may be able to recover financially from the business-cycle component of their post-graduation woes. Although even here, Simkovic and McIntyre acknowledge that past results cannot guarantee future performance. The Great Recession may produce aftershocks that differ from earlier recessions.
All of this, though, edges around the elephant in the room: Have shifts occurred in the legal profession that will make that work less remunerative or less satisfying to law graduates? And/or have changes occurred that will make remunerative, satisfying work available to a smaller percentage of law graduates?
Simkovic and McIntyre have limited data on those questions. Their primary dataset does not yet include anyone who earned a JD after 2008. A supplemental analysis seems to encompass some post-2008 degree holders, but the results are limited. Simkovic and McIntyre remain confident that any structural change will help, rather than hurt, law graduates–but their evidence speaks to that issue only in historical terms at best. What is actually happening in the workplace today?
The Class of 2010
Five years ago, the Class of 2010 was sitting in our classrooms, anticipating graduation, dreading the bar exam, and worrying about finding a job. Did they find jobs? What kind of work are they doing?
I decided to find out by tracking employment results for more than 1,200 graduates from that year. I’ll be releasing that paper later this week, but here’s a preview: the class’s employment pattern has not improved much from where it stood nine months after graduation. The results are strikingly low compared to the Class of 2000 (the one followed by the massive After the JD study). The decline in law firm employment is particularly marked: just 40% of the group I followed works in a law firm of any size, compared to 62.3% for the Class of 2000 at a similar point in their careers.
A change of that magnitude, in the primary sector (law firms) that hires new law graduates, smacks of structural change. I’m not talking just about BigLaw; these changes pervaded the employment landscape. Stay tuned.
Best Practices for the LSAC
February 11th, 2015 / By Deborah J. Merritt
Back in 2012, the California Department of Fair Employment and Housing sued LSAC, claiming that the organization imposed unreasonable requirements on test-takers seeking disability accommodations. The Department of Justice intervened, siding with the plaintiffs. In May 2014, the parties agreed to a consent decree. LSAC agreed to (a) end the practice of flagging scores from individuals who received extended time as an accommodation; (b) pay $7.73 million in civil penalties and compensation to affected individuals; (c) streamline its evaluation of accommodation requests; and (d) implement additional best practices recommended by a panel of experts. For a quick summary of the decree, see this press release.
The expert panel issued their report last month. Four of the panel members have also provided an easy-to-digest (but unofficial) executive summary of the report. The recommendations offer measured, thoughtful approaches that will allow all potential law students to request accommodations in a fair manner. Too often as a society, we purport to accommodate people with differences, but then subject them to cumbersome, demeaning processes to prove their worth. The best practices endorsed by the consent-decree panel will put an end to that for LSAT takers.
The parties to the original lawsuit have until February 26 to notify the other parties if they plan to challenge any portion of the expert report. So far, no one has issued such a notice. Let’s hope that continues and that LSAC swiftly implements these best practices. This is both an effective way to end a lawsuit and a fair way to treat potential colleagues in the legal profession.
If you would like to offer comments to LSAC, including encouragement for them to accept the recommendations and put this lawsuit behind them, feel free to email LSAC’s Excecutive Director Daniel Bernstine at dbernstine@lsac.org.
I’m proud to say that my colleague, Ruth Colker, was a member of this expert panel. She was chosen by the other four panelists (two apiece from each side of the litigation) from a list of three top-notch experts drawn up by the Department of Justice. In her day job, Ruth serves as a Distinguished University Professor of The Ohio State University and Heck Faust Memorial Chair in Constitutional Law at our College of Law.
I Am the Law
January 28th, 2015 / By Deborah J. Merritt
My colleague Kyle McEntee has a new project that you’ll want to check out. “I Am the Law” is a series of podcasts exploring a wide range of law practice jobs. These aren’t typical attorney interviews: the lawyers offer more detail about their practices than I’ve heard on other broadcasts or career panels.
The podcasts are rich in detail, but free for all listeners. Law students will find a wealth of information on practice areas, work settings, and the paths that individual attorneys followed in their careers. I hope that career services offices will recommend the podcasts to their students.
Prospective law students will also appreciate these podcasts. The discussions can take them well beyond the media stereotypes of BigLaw associates and aggressive courtroom lawyers. What’s it like to practice as a family or patent law attorney? How about real estate, immigration, nonprofit management, and transactional work? Can you believe that there is still room for a “writs attorney” in the twenty-first century? These 20-30 minute podcasts are perfect for listening while working out, riding the bus, or walking across campus.
I’m intrigued, finally, about the possibility of using these podcasts to complement doctrinal courses. I wish that when I taught first-year Torts, I could have asked my students to listen to the podcast with personal injury attorney Tricia Dennis. (Disclosure: I’m the host who interviewed Tricia, and I serve as an ongoing host for I Am the Law.) We forget how much of our law school curriculum focuses on appellate lawyering. Even when we ask students to imagine how they would apply a rule to a client’s problem, it’s hard for them to see the world through a practitioner’s eyes.
We should do much more in law school to help students understand their future roles as problem solvers for real people and organizations. But as we explore those avenues, the I Am the Law podcasts are an easy, cost-free way to give students a small taste of law practice related to the subject areas you teach.
Have a listen. I think you’ll be impressed, as I was, by the thoughtfulness of these lawyers in explaining both their current work and their personal paths in the law.
How Deans Should Game the Above the Law Rankings
May 1st, 2014 / By Kyle McEntee
The popular legal news website Above the Law just announced its 2014 Top 50 Law School Rankings. ATL’s methodology focuses exclusively on outcomes: only jobs, total cost, and alumni satisfaction matter.
I generally disfavor rankings. Ranking systems appeal to a desire for clear answers even when clear answers don’t exist. Through a simple list format, rankings project the appearance of authority and value even when they provide neither.
Inherent issues aside, ATL’s rankings at least focus on elements that should and do matter to prospective students. As a result, the ATL rankings incentivize schools to act in ways that measurably help students. That’s a welcome change.
If you’re a law school dean that wants to increase its standing in the ATL rankings, here are the two most critical steps:
1. Lower Tuition
The ATL rankings factor in total educational cost, which combines living expenses, tuition, inflation, and the interest accumulated during law school. Unless a law school moves across the country, student living expenses are relatively inflexible. To compete on the education cost metric schools must either lower tuition or convince ATL to use net price instead of sticker price.
In using sticker price, ATL penalizes schools that use a high tuition, high discount model. That’s basically every school (but maybe changing). Schools that shift to a more transparent pricing model will benefit in next year’s rankings without taking in less tuition revenue.
2. Maintain or Reduce Class Size
Although class sizes are not directly measured by the ATL rankings, each employment metric either controls for graduating class size (SCOTUS clerkships; Article III judges) or relies on an employment percentage for which graduating class size is the denominator. Graduating class size is a function of incoming class size, net transfers, and students dropping out or taking longer to finish school than anticipated.
Smaller incoming classes demonstrate a modicum of social and professional responsibility in a visible manner. This buys trust from incoming students. But the urge to take more transfers to generate more revenue must be appealing these days as schools try to make up for lost 1L revenue. After all, transfers pay more, do not impact LSAT or GPA medians, have low marginal cost, and integrate rather silently. Large transfer classes also seem appealing if you believe that enrollment cuts have been too deep—an increasingly common, yet disturbing belief.
Due to ATL’s methodology, schools cannot hide from enrollment levels that adversely affect employment outcomes. Neither can schools make up for over-enrollment by funding jobs for graduates. As such, resisting the temptation to grow enrollment will benefit schools on rankings that unapologetically penalize schools for graduating too many students into a crowded entry-level market.
Schools game rankings. That’s just a basic fact about modern higher education. At least with ATL’s rankings, gaming the rankings produces measurable, positive results for students and the profession. It sure beats an incentive to burn money on blackacre to secure a higher ranking.
About Law School Cafe
Cafe Manager & Co-Moderator
Deborah J. Merritt
Cafe Designer & Co-Moderator
Kyle McEntee
Law School Cafe is a resource for anyone interested in changes in legal education and the legal profession.
Around the Cafe
Subscribe
Categories
Recent Comments
- on Scholarship Advice
- on ExamSoft: New Evidence from NCBE
- on COVID-19 and the Bar Exam
- on Women Law Students: Still Not Equal
- on Ranking Academic Impact
Recent Posts
- The Bot Takes a Bow
- Fundamental Legal Concepts and Principles
- Lay Down the Law
- The Bot Updates the Bar Exam
- GPT-4 Beats the Bar Exam
Monthly Archives
Participate
Have something you think our audience would like to hear about? Interested in writing one or more guest posts? Send an email to the cafe manager at merritt52@gmail.com. We are interested in publishing posts from practitioners, students, faculty, and industry professionals.