You are currently browsing the archives for Deborah J. Merritt.
What Use Is the BLS?
March 28th, 2015 / By Deborah J. Merritt
What is the Bureau of Labor Statistics (BLS), and what can it do for you? The BLS is an independent statistical agency that measures “labor market activity, working conditions, and price changes in the economy.” You’ve sampled BLS wares if you’ve relied upon the Consumer Price Index, unemmployment rates, or average wages.
One program within BLS tries to project employment growth for hundreds of different occupations. The Bureau issues these forecasts every two years, with each projection spanning a decade. The most recent projections, released in December 2013, attempt to forecast occupational growth between 2012 and 2022.
Why does BLS spend your tax dollars trying to do this? Most parents can’t predict what their teenagers will do next week. How does the BLS think it can predict the behavior of an entire economy, including growth rates in so many different occupations?
The truth is that it can’t, at least not with the level of accuracy that some users would like. There are just too many variables, not to mention acts of god and war. The latest evaluation of BLS’s occupational projections found that, when BLS projected occupational growth rates between 1996 and 2006, it failed to foresee the following:
* Immigration would be higher than the Census Bureau predicted
* Women’s labor force participation would decline
* Terrorists would hijack 4 jets, level the WTC, and damage the Pentagon
* The United States would go to war with both Afganistan and Iraq
* A housing bubble would double home prices over the decade
* Internet-based services would cut the number of travel agents by a third
It was a tumultuous decade, but so are most decades. Given the twists and turns of human history, which affect the type of work that humans do, why does BLS even bother with occupational projections?
Better Than the Alternatives
Like democracy, BLS’s projections seem to be better than the alternatives. In particular, these forecasts are better than ones that rely solely on historical trends. In 2010, the Bureau tested its model against four different “naive models” that drew solely on historical data. A common naive model (and one that the Bureau tested) predicts each occupation’s growth rate based on that occupation’s rate of growth during the previous 10 years. Another variation, also tested by the Bureau, uses the most recent five years to project future growth.
On three out of four measures, the Bureau’s predictions outperformed all of the naive models. Predicting the future is difficult, especially when that future includes human actions. The Bureau’s experience, however, suggests that past performance is not the best guide to occupational growth; adding other ingredients to the forecast improves information.
Who Needs It?
Even if BLS predictions are better than naive models, who needs these predictions? Why engage in such an imprecise exercise? BLS began projecting occupational growth after World War II in order to help returning veterans identify promising career paths. The program persisted as a way to serve “individuals seeking career guidance,” as well as “policymakers, community planners, and educational authorities who need information for long-term policy planning purposes.”
If BLS wants students to use its occupational projections for “career guidance,” then why does it warn against using the projections to predict labor shortages or surpluses? Don’t students examine these projections precisely to determine which occupations are growing and which ones are declining? How is occupational “growth” different from a labor “shortage” in that occupation?
The two concepts are related, yet different. Remember that BLS projects (however imperfectly) the number of people who will actually fill an occupation a decade later. The Bureau doesn’t estimate how many people will want to work in that field or how many will prepare to do so; that’s not its task. The Bureau also assumes that the labor market will “clear.” In other words, if demand falls for workers in a particular field, those workers will go elsewhere. They won’t simply hang around the edges of the occupation, constituting a surplus labor supply.
This doesn’t mean, however, that the number of workers preparing to enter an occupation is irrelevant to predicting job and salary prospects for that occupation. If the pipeline of aspiring workers is easy to quantify, and if the occupation itself is tightly defined, then comparing the worker supply to job projections can yield useful information. If labor supply greatly exceeds likely job openings, then one of three things are likely to happen: (1) some of the workers will take other jobs; (2) wages in the occupation will decline; or (3) both.
What About Law?
The worker pipeline is relatively easy to specify in law. Almost no one becomes a lawyer without obtaining a JD, and there is evidence (p. 72) that most law graduates want to practice law at least for a while. The occupation itself is also well defined. Law graduates can apply their education to a range of law-related jobs, but there is widespread consensus on which jobs are “lawyering” jobs that require bar admission. These are the same jobs that graduates, on the whole, prefer.
Under those conditions, it is useful to compare the number of law school graduates to projected job openings for lawyers. That is what I did several years ago. At that time, the number of students progressing through the law school pipeline greatly exceeded the number of lawyering positions that BLS projected. A substantial number of those graduates, I predicted, would have to find work outside of law practice. Wages for entry-level lawyers might also fall.
That is, in fact, what happened. My recent study of new lawyers admitted to the Ohio bar confirms that, four and a half years after graduation, one quarter of licensed lawyers were working in jobs that did not require a law degree. After accounting for graduates who didn’t take or pass the bar exam, it appears that a full third of recent law school graduates are not practicing as lawyers.
The good news is that my study suggests there may be more job openings for lawyers than BLS projected. Not enough to satisfy all of the graduates who want those jobs, but more than BLS estimated.
Meanwhile, there is also evidence that wages have declined for entry-level lawyers. The median starting salary reported to NALP for the Class of 2008 was $72,000; five years later, the median reported salary for the Class of 2013 was $62,467. The comparison looks even worse after adjusting for inflation: If the median wage had remained at the 2008 level, it would have reached almost $78,000 by 2013. The real median wage for new lawyers fell by 19.8% over those five years.
Will law graduates who were unable to find a lawyering job find satisfaction in other jobs? They might; probably some will and some won’t. Will they prosper financially from their law degree, regardless of occupation? They might, if historical patterns hold. To the extent their wage losses represent effects of the recession, will they make up those differences later in their careers? Again, they might if historical patterns hold. But for students investing more than $100,000 in a legal education, it’s worth considering as much information as possible. That includes BLS projections for their desired occupation.
These projections are also useful–when combined with other available information–for legal educators to consider. The career prospects of our graduates should inform the educational programs we design, as well as the information we offer potential applicants. BLS projections represent only a small piece of this puzzle, but they offer one perspective on how the labor market for lawyers is performing.
What About Those New Projections?
The BLS recently changed the way in which it measures occupational “separations.” That’s an estimate of the number of people who will leave a particular occupation. This measure, in turn, affects the projection of job openings; when a worker leaves an occupation, that departure often creates a job opening. Under this new method, BLS will project more lawyering jobs than it did in the past. That sounds like good news for aspiring lawyers, and it is–in part. The change also reveals some unsettling trends in our profession, which I’ll explore in a future post.
ExamSoft: By the Numbers
March 26th, 2015 / By Deborah J. Merritt
Earlier this week I explained why the ExamSoft fiasco could have lowered bar passage rates in most states, including some states that did not use the software. But did it happen that way? Only ExamSoft and the National Conference of Bar Examiners have the data that will tell us for sure. But here’s a strong piece of supporting evidence:
Among states that did not experience the ExamSoft crisis, the average bar passage rate for first-time takers from ABA-accredited law schools fell from 81% in July 2013 to 78% in July 2014. That’s a drop of 3 percentage points.
Among the states that were exposed to the ExamSoft problems, the average bar passage rate for the same group fell from 83% in July 2013 to 78% in July 2014. That’s a 5 point drop, two percentage points more than the drop in the “unaffected” states.
Derek Muller did the important work of distinguishing these two groups of states. Like him, I count a state as an “ExamSoft” one if it used that software company and its exam takers wrote their essays on July 29 (the day of the upload crisis). There are 40 states in that group. The unaffected states are the other 10 plus the District of Columbia; these jurisdictions either did not contract with ExamSoft or their examinees wrote essays on a different day.
The comparison between these two groups is powerful. What, other than the ExamSoft debacle could account for the difference between the two? A 2-point difference is not one that occurs by chance in a population this size. I checked and the probability of this happening by chance (that is, by separating the states randomly into two groups of this size) is so small that it registered as 0.00 on my probability calculator.
It’s also hard to imagine another factor that would explain the difference. What do Arizona, DC, Kentucky, Louisiana, Maine, Massachusetts, Nebraska, New Jersey, Virginia, Wisconsin, and Wyoming have in common other than that their test takers were not directly affected by ExamSoft’s malfunction? Large states and small states; Eastern states and Western states; red states and blue states.
Of course, as I explained in my previous post, examinees in 10 of those 11 jurisdictions ultimately suffered from the glitch; that effect came through the equating and scaling process. The only jurisdiction that escaped completely was Louisiana, which used neither ExamSoft nor the MBE. That state, by the way, enjoyed a large increase in its bar passage rate between July 2013 and July 2014.
This is scary on at least four levels:
1. The ExamSoft breakdown affected performance sufficiently that states using the software suffered an average drop of 2 percentage points in bar passage.
2. The equating and scaling process amplified the drop in raw scores. These processes dropped pass rates as much as three more percentage points across the nation. In states where raw scores were affected, pass rates fell an average of 5 percentage points. In other states, the pass rate fell an average of 3 percentage points. (I say “as much as” here because it is possible that other factors account for some of this drop; my comparison can’t control for that possibility. It seems clear, however, that equating and scaling amplified the raw-score drop and accounted for some–perhaps all–of this drop.)
3. Hundreds of test takers–probably more than 1,500 nationwide–failed the bar exam when they should have passed.
4. ExamSoft and NCBE have been completely unresponsive to this problem, despite the fact that these data have been available to them.
One final note: the comparisons in this post are a conservative test of the ExamSoft hypothesis, because I created a simple dichotomy between states exposed directly to the upload failure and those with no direct exposure. It is quite likely that states in the first group differed in the extent to which their examinees suffered. In some states, most test takers may have successfully uploaded their essays on the first try; in others, a large percentage of examinees may have struggled for hours. Those differences could account for variations within the “ExamSoft” states.
ExamSoft and NCBE could make those more nuanced distinctions. From the available data, however, there seems little doubt that the ExamSoft wreck seriously affected results of the July 2014 bar exam.
* I am grateful to Amy Otto, a former student who is wise in the way of statistics and who helped me think through these analyses.
ExamSoft After All?
March 24th, 2015 / By Deborah J. Merritt
Why did so many people fail the July 2014 bar exam? Among graduates of ABA-accredited law schools who took the exam for the first time last summer, just 78% passed. A year earlier, in July 2013, 82% passed. What explains a four-point drop in a single year?
The ExamSoft debacle looked like an obvious culprit. Time wasted, increased anxiety, and loss of sleep could have affected the performance of some test takers. For those examinees, even a few points might have spelled the difference between success and failure.
Thoughtful analyses, however, pointed out that pass rates fell even in states that did not use ExamSoft. What, then, explains such a large performance drop across so many states? After looking closely at the way in which NCBE and states grade the bar exam, I’ve concluded that ExamSoft probably was the major culprit. Let me explain why–including the impact on test takers in states that didn’t use ExamSoft–by walking you through the process step by step. Here’s how it could have happened:
Tuesday, July 29, 2014
Bar exam takers in about forty states finished the essay portion of the exam and attempted to upload their answers through ExamSoft. But for some number of them, the essays wouldn’t upload. We don’t know the exact number of affected exam takers, but it seems to have been quite large. ExamSoft admitted to a “six-hour backlog” and at least sixteen states ultimately extended their submission deadlines.
Meanwhile, these exam takers were trying to upload their exams, calling customer service, and worrying about the issue (wouldn’t you, if failure to upload meant bar failure?) instead of eating dinner, reviewing their notes for the next day’s MBE, and getting to sleep.
Wednesday, July 30, 2014
Test takers in every state but Louisiana took the multiple choice MBE. In some states, no one had been affected by the upload problem. In others, lots of people were. They were tired, stressed, and had spent less time reviewing. Let’s suppose that, due to these issues, the ExamSoft victims performed somewhat less well than they would have performed under normal conditions. Instead of answering 129 questions correctly (a typical raw score for the July MBE), they answered just 125 questions correctly.
August 2014: Equating
The National Conference of Bar Examiners (NCBE) received all of the MBE answers and began to process them. The raw scores for ExamSoft victims were lower than those for typical July examinees, and those scores affected the mean for the entire pool. Most important, mean scores were lower for both the “control questions” and other questions. “Control questions” is my own shorthand for a key group of questions; these are questions that have appeared on previous bar exams, as well as the most current one. By analyzing scores for the control questions (both past and present) and new questions, NCBE can tell whether one group of exam takers is more or less able than an earlier group. For a more detailed explanation of the process, see this article.
These control questions serve an important function; they allow NCBE to “equate” exam difficulty over time. What if the Evidence questions one year are harder than those for the previous year? Pass rates would fall because of an unfairly hard exam, not because of any difference in the exam takers’ ability. By analyzing responses to the control questions (compared to previous years) and the new questions, NCBE can detect changes in exam difficulty and adjust raw scores to account for them.
Conversely, these analyses can confirm that lower scores on an exam stem from the examinees’ lower ability rather than any change in the exam difficulty. Weak performance on control questions will signal that the examinees are “less able” than previous groups of examinees.
But here’s the rub: NCBE can’t tell from this general analysis why a group of examinees is less able than an earlier group. Most of the time, we would assume that “less able” means less innately talented, less well prepared, or less motivated. But “less able” can also mean distracted, stressed, and tired because of a massive software crash the night before. Anything that affects performance of a large number of test takers, even if the individual impact is relatively small, will make the group appear “less able” in the equating process that NCBE performs.
That’s step one of my theory: struggling with ExamSoft made a large number of July 2014 examinees perform somewhat below their real ability level. Those lower scores, in turn, lowered the overall performance level of the group–especially when compared, through the control questions, to earlier groups of examinees. If thousands of examinees went out partying the night before the July 2014 MBE, no one would be surprised if the group as a whole produced a lower mean score. That’s what happened here–except that the examinees were frantically trying to upload essay questions rather than partying.
August 2014: Scaling
Once NCBE determines the ability level of a group of examinees, as well as the relative difficulty of the test, it adjusts the raw scores to account for these factors. The adjustment process is called “scaling” and it consists of adding points to the examinees’ raw scores. In a year with an easy test or “less able” examinees, the scaling process adds just a few points to each examinee’s raw score. Groups who faced a harder test or were “more able,” get more points. [Note that the process is a little more complicated that this; each examinee doesn’t get exactly the same point addition. The general process, however, works in this way–and affects the score of every single examinee. See this article for more.]
This is the point at which the ExamSoft crisis started to affect all examinees. NCBE doesn’t scale scores just for test takers who seem less able than others; it scales scores for the entire group. The mean scaled score for the July 2014 MBE was 141.5, almost three points lower than the mean scaled score in July 2013 (which was 144.3). This was also the lowest scaled score in ten years. See this report (p. 35) for a table reporting those scores.
It’s essential to remember that the scaling process affects every examinee in every state that uses the MBE. Test takers in states unaffected by ExamSoft got raw scores that reflected their ability, but they got a smaller scaling increment than they would have received without ExamSoft depressing outcomes in other states. The direct ExamSoft victims, of course, suffered a double whammy: they obtained a lower raw score than they might have otherwise achieved, plus a lower scaling boost to that score.
Fall 2014: Essay Scoring
After NCBE finished calculating and scaling MBE scores, the action moved to the states. States (except for Louisiana, which doesn’t use the MBE), incorporated the artificially depressed MBE scores into their bar score formulas. Remember that those MBE scores were lower for every exam taker than they would have been without the ExamSoft effect.
The damage, though, didn’t stop there. Many (perhaps most) states scale the raw scores from their essay exams to MBE scores. Here’s an article that explains the process in fairly simple terms, and I’ll attempt to sum it up here.
Scaling takes raw essay scores and arranges them on a skeleton provided by that state’s scaled MBE results. When the process is done, the mean essay score will be the same as the mean scaled MBE score for that state. The standard deviations for both will also be the same.
What does that mean in everyday English? It means that your state’s scaled MBE scores determine the grading curve for the essays. If test takers in your state bombed the MBE, they will all get lower scores on the essays as well. If they aced the MBE, they’ll get higher essay scores.
Note that this scaling process is a group-wide one, not an individual one. An individual who bombed the MBE won’t necessarily flunk the essays as well. Scaling uses indicators of group performance to adjust essay scores for the group as a whole. The exam taker who wrote the best set of essays in a state will still get the highest essay score in that state; her scaled score just won’t be as high as it would have been if her fellow test takers had done better on the MBE.
Scaling raw essay scores, like scaling the raw MBE scores, produces good results in most years. If one year’s graders have a snit and give everyone low scores on the essay part of the exam, the scaling process will say, “wait a minute, the MBE scores show that this group of test takers is just as good as last year’s. We need to pull up the essay scores to mirror performance on the MBE.” Conversely, if the graders are too generous (or the essay questions were too easy), the scaling process will say “Uh-oh. The MBE scores show us that this year’s group is no better than last year’s. We need to pull down your scores to keep them in line with what previous graders have done.”
The scaled MBE scores in July 2014 told the states: “Your test takers weren’t as good this year as last year. Pull down those essay scores.” Once again, this scaling process affected everyone who took the bar exam in a state that uses the MBE and scales essays to the MBE. I don’t know how many states are in the latter camp, but NCBE strongly encourages states to scale their essay scores.
Fall 2014: MPT Scoring
You guessed it. States also scale MPT scores to the MBE. Once again, MBE scores told them that this group of exam takers was “less able” than earlier groups so they should scale down MPT scores. That would have happened in every state that uses both the MBE and MPT, and scales the latter scores to the former.
Conclusion
So there you have it: this is how poor performance by ExamSoft victims could have depressed scores for exam takers nationwide. For every exam taker (except those in Louisiana) there was at least a single hit: a lower scaled MBE score. For many exam takers there were three hits: lower scaled MBE score, lower scaled essay score, and lower scaled MPT score. For some direct victims of the ExamSoft crisis, there was yet a fourth hit: a lower raw score on the MBE. But, as I hope I’ve shown here, those raw scores were also pebbles that set off much larger ripples in the pond of bar results. If you throw enough pebbles into a pond all at once, you trigger a pretty big wave.
Erica Moeser, the NCBE President, has defended the July 2014 exam results on the ground that test takers were “less able” than earlier groups of test takers. She’s correct in the limited sense that the national group of test takers performed less well, on average, on the MBE than the national group did in previous years. But, unless NCBE has done more sophisticated analyses of state-by-state raw scores, that doesn’t tell us why the exam takers performed less “ably.”
Law deans like Brooklyn’s Nick Allard are clearly right that we need a more thorough investigation of the July 2014 bar results. It’s too late to make whole the 2,300 or so test takers who may have unfairly failed the exam. They’ve already grappled with a profound sense of failure, lost jobs, studied for and taken the February exam, or given up on a career practicing law. There may, though, be some way to offer them redress–at least the knowledge that they were subject to an unfair process. We need to unravel the mystery of July 2014, both to make any possible amends and to protect law graduates in the future.
I plan to post some more thoughts on this, including some suggestions about how NCBE (or a neutral outsider) could further examine the July 2014 results. Meanwhile, please let me know if you have thoughts on my analysis. I’m not a bar exam insider, although I studied some of these issues once before. This is complicated stuff, and I welcome any comments or corrections.
Updated on September 21, 2015, to correct reported pass rates.
Needing Law Schools
March 22nd, 2015 / By Deborah J. Merritt
I agree entirely with Noah Feldman that society needs law schools. He couldn’t have said it better. This, however, is exactly why law schools need to fix their financial model. Most schools lack the big endowments of Harvard and other elite schools. Students, meanwhile, are increasingly unwilling to pay so much more tuition than Feldman did in the 1990’s or I did in the 1970’s. We need to keep asking: Why does it cost so much more today to learn what the law “can be”?
I learned a lot about what the law can be from Ruth Bader Ginsburg, my constitutional law professor at Columbia in 1979. I also learned from Herbert Wechsler, author of the much-cited article on “neutral principles” in constitutional law; William Carey, one of the New Deal architects and an early chair of the SEC; E. Allan Farnsworth, Reporter for the Restatement (Second) of Contracts; Maurice Rosenberg, one of the earliest legal scholars to apply social science research to legal problems; and many others. Why were all of these luminaries able to teach me and my classmates for so much less tuition than Columbia and other schools demand today?
In part, they earned less. I know that, because I am the daughter of yet another Columbia professor from that era: William K. (“Ken”) Jones. Our family did just fine financially, but we didn’t have the affluence that law professors enjoy today. Another explanation rests on the enormous number of staff members that law schools now need to operate. Communications staff, admissions staff, development staff, student services staff . . . . Each seems indispensable in the modern law school, but how many contribute to our mission of teaching students and others what the law can be?
I doubt that it’s possible to unwind the contemporary law school, to dismiss all of the staff, and go back to an earlier, simple world. Although it’s a charming notion, isn’t it? We could simply post our lower tuition, admit students who apply (without spending time marketing to them), teach them, and send them into the world knowing something about both what the law is and what it can be. Meanwhile, we would publish and engage in law reform efforts–as Ginsburg, Wechsler, Carey, and the others did–while teaching four courses a year.
I know that’s unlikely to happen, so we’ve got to find other ways to fix the financial model. Shifting the first year of law school to the undergraduate curriculum makes sense to me. Let’s teach more people about both the power of law and what it can be. Meanwhile, let’s lower tuition for those who will actually practice law. We, as professors, can teach people what the law can be–but our graduates are essential to make those changes happen.
How Many Lawyers?
March 20th, 2015 / By Deborah J. Merritt
A few years ago, I used employment projections from the Bureau of Labor Statistics to project the number of job openings for licensed lawyers during the current decade. At the time, BLS was the best available source for that type of projection; it remains a useful resource today. The BLS makes these predictions precisely to help workers, employers, and policymakers understand the likely demand for workers in particular occupations.
Why should law schools care about these predictions? As Michael Simkovic and Frank McIntyre show in two recent papers, a JD historically has conferred financial advantage (compared to entering the workforce with just a BA) even if graduates did not work as practicing lawyers. If law graduates reap financial returns from their degrees, regardless of the jobs they take, does it matter how many jobs they find as practicing lawyers?
Some scholars (including me) wonder whether financial returns to a law degree will remain as high as they have been. But let’s leave that debate aside for now. Let’s consider, instead, why law schools should care about the number of grads who will find jobs as licensed lawyers–and what that number might be.
Who Cares?
Law graduates, students, and prospective students seem to care. Several sources indicates that, although some graduates enjoy work that does not require bar admission, graduates overall prefer to practice law. Every year, for example, NALP publishes a table showing how many graduates are employed but still seeking work. These tables consistently show that grads in jobs that require bar admission are far less likely than other grads to be seeking new jobs. In 2010, for example, just 15.1% of the graduates in “bar required” work were seeking other employment, while 48.1% of those in “JD preferred” jobs and 49.0% of grads in “other professional positions” were doing so. [Sorry, source not available online]
Similar preferences emerge later in the career. The After the JD longitudinal study found that law grads working in non-lawyering jobs were less satisfied than those in lawyering positions. The distinction held both seven and twelve years after law school graduation. [No online link for that one either, I’m afraid, but check p. 70 of the AJD II and AJD III reports.]
If graduates care, I suspect that students and prospective students care as well. And if only out of self interest, law schools should care too. We can tell students that a law degree offers many options, and that historically those options have paid off financially compared to a BA alone, but the current crop of students isn’t buying that pitch. If they want to practice law, they’re still coming to law school. But if they’re looking for intellectual stimulation, a degree with diverse options, or other benefits, they seem to be going elsewhere.
This is partly why I think law schools should examine the way we package the education we offer. We’re good at teaching close reading, careful writing, and critical thinking–and we could teach those skills to undergraduates. Talented undergrads want those skills, and employers will pay a premium for them. At the same time, we’re good (and could be better) at teaching advanced legal doctrine and other intellectual skills essential for law practice. We can keep providing that education to a more focused group of JD students who primarily will become lawyers.
So How Many Lawyers?
This brings me back to the question I started with: If prospective law students care about whether they will get jobs as licensed attorneys, and if law schools should care about that question (if only to attract students), about how many law graduates are able to get jobs that require bar admission?
My recent study of law graduates in the State of Ohio gave me some numbers to work with. I tracked job outcomes for all 1,214 new lawyers who passed the Ohio bar exam in 2010. In December 2014, about 75% of them held jobs that required a law license. Most of the rest were employed, but in other types of work.
My population included only licensed lawyers, not law graduates who didn’t take or pass a bar exam. One thing we know about the latter group is that they can’t be practicing law. So, after performing some calculations to account for that group, I estimated that about two-thirds of 2010 graduates were practicing law four and a half years after graduation.
Will the same percentage hold for graduates from other years and in other states? I don’t know; we often have to deal with limited data and isolated points of reference in making real-world plans. I explain in the paper why I think Ohio offers a useful perspective, and why I think the annual number of new bar-required jobs will remain stable in coming years. I’ll write more about both of those issues here soon.
The Good News
The good news for both law schools and prospective students is that my estimate is higher than the BLS’s historic projections. I estimate that, four and a half years after graduation, about 29,250 members of the Class of 2010 are practicing law. Some of the jobs are dubious solo practices; some undoubtedly are part-time, temporary, and/or low paid. Some of them are jobs that a graduate secured only after failing and re-taking the bar exam. Prospective students need to take those factors into account, not simply consider whether they’ll be able to find a job practicing law.
When all is said and done, though, graduates are finding more “lawyering” jobs than BLS once predicted–although not as many as BLS predicts through a proposed revision to its forecasting method. I’ll comment on the latter in another post.
The other piece of good news is that, if my calculations are correct, and if law school attrition rates remain constant, then about 84.5% of the current 1L class will find lawyering jobs within a few years after graduation. That level of job placement may be sufficiently attractive to maintain enrollment at current rates.
The Bad News
I couched that last sentence carefully: “to maintain enrollment at current rates.” Better placement in lawyering jobs will reassure prospective law students, but I doubt it will draw them back to law school in droves. Law schools, meanwhile, will need to worry about bar passage rates as they enroll students with lower credentials. Declining bar passage rates will discourage potential applicants both directly and indirectly, as they depress the percentage of graduates working in jobs that require bar passage.
Schools will also vary in the percentage of graduates they place in jobs requiring bar admission. Some will place more than 85% in those positions; others will place much less. If I’m right that potential students care about getting jobs that require a law license, enrollment declines will continue at the latter schools.
Summing Up
In making predictions, both here and in my paper, I offer some very specific numbers. I do that to offer a point of reference for debate; I can’t say exactly how many members of the Class of 2017 will find lawyering jobs, or how many students will apply to law school in 2018. I do think, though, that JD students care about their odds of securing a job that requires a law license and that law schools need to account for that preference. To do that, it helps to know as much as we can about operation of the legal market.
Bankruptcy Petitions
March 17th, 2015 / By Deborah J. Merritt
Michael Sousa has written a thoughtful paper about the issue of non-lawyers preparing bankruptcy petitions for consumer debtors. As Sousa acknowledges, some of these preparers urge their clients to commit fraud; others serve their clients incompetently. But then again, the same is true of some bankruptcy lawyers–and the lawyers charge more than many consumers can afford.
After reviewing the field, Sousa proposes creation of a cadre of licensed petition preparers for no-asset Chapter 7 bankruptcy filings. The United States Trustee’s Office could regulate these practitioners, requiring them to demonstrate adequate knowledge of the law and adherence to appropriate ethical standards. The practitioners, however, would not need JD’s or law licenses.
Sousa’s proposal makes sense to me. I note it here because he has identified yet another area in which non-lawyers may be able to handle the law-related needs of clients who cannot afford an attorney’s fee. As Sousa details, the federal government already allows qualified non-lawyers to represent clients before the IRS, US Patent and Trademark Office, and (in immigration matters) the Department of Homeland Security. The transfer of law-related business to non-lawyers has been slow but steady.
What Happened to the Class of 2010?
March 14th, 2015 / By Deborah J. Merritt
It has been almost five years since the Class of 2010 walked across the stage to pick up their JDs. Since then, we’ve debated whether the weak job outcomes for that class stemmed solely from the recession or represented structural changes within the profession itself. We’ve also wondered whether the graduates would improve their employment status as they moved into the workforce and the economy picked up. I decided to find out.
1,214 New Lawyers
In a paper just posted to SSRN, I tracked job outcomes for all 1,214 new lawyers who passed the Ohio bar exam in 2010 and then joined the bar. The study doesn’t focus on the graduates of a particular school, as some analyses do. I included all bar-passers, regardless of the school they attended (although note that Ohio only allows graduates of ABA-accredited schools to take the exam). I also followed successful examinees even if they left Ohio.
My focus on bar admittees parallels the approach taken in several earlier studies of the legal profession, including the recent After the JD (AJD) surveys. As I note in the paper, that means I can’t say very much about the approximately 12% of law school graduates who never gain bar admission. It also means that I overstate the percentage of jobs that require bar admission, although I offer some corrections for this in the paper.
I also explain in the paper why Ohio offers a useful perspective on trends nationally. The state’s 9-month outcomes, as reported by NALP, are similar to national averages. Ohio is the ninth largest state for both number of practicing lawyers and number of jobs taken by new law graduates. It’s also home to the nation’s largest law firm and several other BigLaw shops. And the state has a healthy economy, with an overall unemployment rate (4.8% in December 2014) that is lower than in states with the largest legal markets. During the same month, New York’s unemployment rate was 5.8%; Illinois, 6.2%; California, 7.0%; and DC, 7.3%.
1,137 Jobs
For each of the 1,214 lawyers in my study, I searched for the job held in December 2014. I didn’t use surveys; I relied entirely on public online sources. This turned out to be much easier than I thought it would be–and produced a much more complete dataset than surveys do. With Ohio’s bar directory, employer websites, directories like LinkedIn, and other sources (e.g., alumni stories published online by law schools), I found a December 2014 job for 1,137 graduates or 93.7% of the population. I explain in the paper why the remaining 6.3% are most likely unemployed (for some, I found direct evidence of that) or serve as a proxy for the percentage unemployed.
Enough of the preliminaries. What did I find? First, it’s hard to read the job histories without feeling great sadness for the Class of 2010. Sure, many of them have ended up in decent jobs. Some may even hold their dream job. But they’ve had (and are still having) a very tough time. Almost a tenth bill themselves as “solo practitioners,” although I found no evidence of an active practice (such as a website) for half of them. At best, these solos are struggling to establish themselves at an early stage of their careers. At worst, they’re unemployed job seekers doing occasional legal work for friends.
Meanwhile, two-thirds of the graduates have switched jobs at least once in four years–that’s twice the rate of turnover that AJD reported for the Class of 2000 during their early careers. Among those who changed jobs, the average number of jobs was 2.7–almost three jobs in just four years. Before you shake your head over those peripatetic millennials, note that median job tenure for 25-34-year-olds nationally is 3.0 years; the millennials actually stick to their jobs somewhat longer than young adults did before 2010.
It’s easy to talk about the job market from the comfort of well paid, tenured positions. It’s a lot harder to be out there in the trenches. Before we talk more about numbers, let’s recognize the fortitude (and ongoing struggles) of the people in the Class of 2010 and other recent classes.
Structural Change
I will write a series of posts highlighting different parts of my findings, but here’s the bottom line: I found strong evidence of structural changes in the employment market. Here are the top indicators:
1. The Class of 2010 graduated almost five years ago, a year after the Great Recession officially ended. It has been a slow and jobless recovery, but there have been signs of growth for quite some time now. There are certainly clients hiring lawyers: the top-earning lawyers are doing very nicely, and a wide range of alternative-service providers are prospering. Yet recent grads are lagging in job outcomes when compared to earlier classes at a similar career point.
2. Only three-quarters of the licensed lawyers hold jobs that require a law license. After adjusting for graduates who never took or passed the bar, I estimate that no more than two-thirds of 2010 graduates work in jobs that require bar admission. We can argue about whether future students will be willing to invest in law school for early-career jobs that don’t use their full education and licensing (I don’t believe they will). It’s clear, however, that the percentage of recent law graduates practicing law has been declining for some time. Whether that makes you smile or weep, it’s a shift in the market.
3. Only 40.4% of the licensed graduates work in law firms, a barely perceptible increase over the 39.5% of the class who reported those jobs nine months after graduation. In fact, the 40.4% almost certainly masks a decrease in law firm employment, because I obtained data only on licensed lawyers; the 9-month figure includes all law school graduates. Even at 40.4%, this is a striking figure. Despite four years of experience and economic recovery, the Class of 2010 made no headway in securing law firm jobs.
4. The low level of law firm employment is even more remarkable when compared to outcomes for the Class of 2000, which was tracked by the AJD study. That class graduated into relatively good economic times, but weathered a recession and jobless recovery during their first years in the profession. When the class reported their three-year outcomes to AJD in late 2002 and 2003, national unemployment levels were actually higher than when I identified jobs held by the Class of 2010 in December 2014. Yet 62.1% of the Class of 2000 practiced law with a firm three years after graduation–compared to just 40.4% of the Class of 2010 four years out. That’s a phenomenal drop of almost twenty-two percentage points.
5. The shift in law firm employment was not due solely (or perhaps even primarily) to changes in BigLaw employment patterns. Ohio, like most states, is not home to a law school ranked among the US Not-News top twenty. Yet we have all of those tasty BigLaw offices and all types of other employers. After the crash, did elite law school graduates jump to Ohio, taking our best jobs and creating a market cascade? I once thought that might be the case, but the evidence says otherwise. The number of elite school graduates passing the Ohio bar actually peaked in 2007–and was low even then. The changes we are seeing in lawyer employment are systemic; they arise from shifts at many levels of the job market.
There’s More
Much more, but I’ll save those data for another day. Meanwhile, you’ll find the full paper here.
Scholarship: Cost and Value
March 12th, 2015 / By Deborah J. Merritt
Critics of legal education raise two key questions about our scholarship: (1) How much value does it offer? And, (2) do law schools have to spend so much money to produce that value?
The answer to the second question is easy: No. We used to produce plenty of superb scholarship with typewriters and four-course teaching loads. Now that we have laptops, tablets, high-powered statistical software, and 24/7 online libraries, our productivity has leaped. Law schools could easily restore teaching loads to four courses a year while still facilitating plenty of good research. The resulting reduction in faculty size could help fund scholarships and reduce tuition.
The answer to the value question is harder. Do we mean immediate pay-off or long term influence? Do we care about value to judges, legislators, practicing attorneys, clients, teachers, students, or some other group? Does each article have to demonstrate value? Or do we recognize that trial and error is part of scholarship as well as other endeavors?
Those are difficult questions and they deserve a series of posts. For now, I’ll limit my discussion to a recent paper by Jeffrey Harrison and Amy Mashburn, which has already provoked considerable commentary. I agree with some of Harrison and Mashburn’s observations, but the empirical part of their paper goes badly astray. Without better method, their conclusions can’t stand. In fact, as I note below, some of their findings seem at odds with their recommendations.
Measuring Citation Strength
Harrison and Mashburn decided to measure the strength of citations to scholarly work, rather than simply count the number of citations. That was an excellent idea; scholars in other fields have done this for decades. There’s a good review of that earlier work in Bornmann & Daniel, Do Citation Counts Measure? A Review of Studies on Citing Behavior, 64 Journal of Documentation 45 (2008). (By the way, isn’t that an amazing name for a journal?)
If Harrison and Mashburn had consulted this literature, they would have found some good guideposts for their own approach. Instead, the paper’s method will make any social scientist cringe. There’s a “control group” that is nothing of the sort, and the method used for choosing articles in that group is badly flawed.* There is little explanation of how they developed or applied their typology (written protocol? inter-rater agreement? training periods?). Harrison and Mashburn tell us only that the distinctions were “highly subjective,” the lines were “difficult to draw,” and “even a second analysis by the current researchers could result in a different count.” Ouch.
Is it possible to make qualitative decisions about citation strength in a thoughtful, documented way? Absolutely. Here’s an example of a recent study of citation types that articulates a rigorous method: Stefan Stremersch, et al., Unraveling Scientific Impact: Citation Types in Marketing Journals, 32 Int’l Journal of Research in Marketing 64 (2015). Harrison and Mashburn might choose a different design than previous scholars, but they need to develop their parameters, articulate them to others, and apply them in a controlled way.
Influence and Usefulness
Harrison and Mashburn conclude that most legal scholarship “is not regarded as useful.” Even when a judge or scholar cites an article, they find, most of the cited articles “serve no useful function in helping the citing author advance or articulate a new idea, theory or insight.” Application of this standard, however, leads to some troubling results.
The authors point, for example, to an article by John Blume, Ten Years of Payne: Victim Impact Evidence in Capital Cases, 88 Cornell L. Rev. 257 (2003). A court cited this article for the seemingly banal fact that “the federal government, the military, and thirty-three of the thirty-eight states with the death penalty have authorized the use of victim impact evidence in capital sentencing.” Harrison and Mashburn dismiss this citation as “solely to the descriptive elements of the article.”
That’s true in a way, but this particular “description” didn’t exist until Blume researched all of that state and federal law to create it. The court wanted to know the state of the law, and Blume provided the answer. This answer may not have “advance[d] . . . a new idea, theory or insight,” but most cases don’t require that level of theory. Disputes do require information about the existing state of the law and Blume assembled information that helped advance resolution of this dispute. Why isn’t that a worthwhile type of influence?
I suspect that judges and practitioners appreciate the type of survey that Blume provided; analyzing the law of 40 jurisdictions requires both time and professional judgment. Blume, of course, did more than just survey the law: he also pointed out crevices and problems in the existing law. But dismissing a citation to the survey portion of his article seems contrary to the authors’ desire to create scholarship that will be more useful.
A reworked method might well distinguish citations to descriptive/survey research from those that adopt a scholar’s new theory. Asking scholars to limit their work to the latter, however, seems counter productive. A lot of people need to know what the law is, not just what it might be.
Judges and Scholars
One statistic in the Harrison and Mashburn article blew me away. On page 25, they note that 73 out of 198 articles from their “top 100” group of journals were cited by courts. That’s more than a third (36.9%) of the articles! I find that a phenomenally high citation rate. I know from personal experience that judges do pay attention to law review articles. When I clerked for Justice O’Connor, for example, she asked us to give her a shelf of law review articles for each of the bench memos we wrote. She didn’t want just our summaries of the articles–she wanted the articles themselves.
But I never would have guessed that the judicial citation rate was as high as 36.9% for professional articles, even for journals from the top 100 schools. At least in judicial circles, there’s a big drop-off between learning from an article and citing the article. Most judges try to keep their opinions lean, and there’s no cultural pressure to cite scholarly works.
I’m not sure how to mesh the judicial citation statistic with the tone of Harrison and Mashburn’s article. More than a third sounds like a high citation rate to me–as does the one quarter figure for journals in the 15-100 group.
Ongoing Discussion
Harrison and Mashburn urge critical debate over the value and funding of legal scholarship, and I back them all the way on that. I wrote this post in that spirit. As I note above, I don’t think law schools need to spend as much money as they do to produce strong levels of excellent scholarship. I also applaud efforts to replace citation counting with more nuanced measures of scholarly value. But we need much stronger empirical work to examine claims like the ones advanced in this paper. Are Harrison and Mashburn right that most legal scholarship “is not regarded as useful”? I don’t know, but I was put off by strong statements with weak empirical evidence.
__________________________
* Harrison and Mashburn chose the first article from each volume. That’s a textbook example of non-random selection: the first article in a volume almost certainly differs, on average, from other articles.
Timing Law School
March 10th, 2015 / By Deborah J. Merritt
Michael Simkovic and Frank McIntyre have a new paper analyzing historic income data for law school graduates. In this article, a supplement to their earlier paper on the lifetime value of a law degree, Simkovic and McIntyre conclude that graduates reap most of the value of a JD whether they graduate in good economic times or poor ones. (Simkovic, by the way, just won an ALI Young Scholar Medal. Congratulations, Mike!)
Simkovic and McIntyre’s latest analyses, they hope, will reassure graduates who earned their degrees in recent years. If history repeats, then these JDs will reap as much financial benefit over their lifetimes as those in previous generations. Simkovic and McIntyre also warn prospective students against trying to “time” law school. It’s difficult to estimate business cycles several years in advance, when a 0L must decide whether to take the plunge. And, again according to historical data, timing won’t make much difference. Under most circumstances, delay will cost more financially than any reward that successful timing could confer.
But Is This Time Different?
History does repeat, at least in the sense of economic conditions that cycle from good to bad and back again. There’s no doubt that recent law school graduates have suffered poor job outcomes partly because of the Great Recession and slow recovery. It’s good to know that graduates may be able to recover financially from the business-cycle component of their post-graduation woes. Although even here, Simkovic and McIntyre acknowledge that past results cannot guarantee future performance. The Great Recession may produce aftershocks that differ from earlier recessions.
All of this, though, edges around the elephant in the room: Have shifts occurred in the legal profession that will make that work less remunerative or less satisfying to law graduates? And/or have changes occurred that will make remunerative, satisfying work available to a smaller percentage of law graduates?
Simkovic and McIntyre have limited data on those questions. Their primary dataset does not yet include anyone who earned a JD after 2008. A supplemental analysis seems to encompass some post-2008 degree holders, but the results are limited. Simkovic and McIntyre remain confident that any structural change will help, rather than hurt, law graduates–but their evidence speaks to that issue only in historical terms at best. What is actually happening in the workplace today?
The Class of 2010
Five years ago, the Class of 2010 was sitting in our classrooms, anticipating graduation, dreading the bar exam, and worrying about finding a job. Did they find jobs? What kind of work are they doing?
I decided to find out by tracking employment results for more than 1,200 graduates from that year. I’ll be releasing that paper later this week, but here’s a preview: the class’s employment pattern has not improved much from where it stood nine months after graduation. The results are strikingly low compared to the Class of 2000 (the one followed by the massive After the JD study). The decline in law firm employment is particularly marked: just 40% of the group I followed works in a law firm of any size, compared to 62.3% for the Class of 2000 at a similar point in their careers.
A change of that magnitude, in the primary sector (law firms) that hires new law graduates, smacks of structural change. I’m not talking just about BigLaw; these changes pervaded the employment landscape. Stay tuned.
Best Practices for the LSAC
February 11th, 2015 / By Deborah J. Merritt
Back in 2012, the California Department of Fair Employment and Housing sued LSAC, claiming that the organization imposed unreasonable requirements on test-takers seeking disability accommodations. The Department of Justice intervened, siding with the plaintiffs. In May 2014, the parties agreed to a consent decree. LSAC agreed to (a) end the practice of flagging scores from individuals who received extended time as an accommodation; (b) pay $7.73 million in civil penalties and compensation to affected individuals; (c) streamline its evaluation of accommodation requests; and (d) implement additional best practices recommended by a panel of experts. For a quick summary of the decree, see this press release.
The expert panel issued their report last month. Four of the panel members have also provided an easy-to-digest (but unofficial) executive summary of the report. The recommendations offer measured, thoughtful approaches that will allow all potential law students to request accommodations in a fair manner. Too often as a society, we purport to accommodate people with differences, but then subject them to cumbersome, demeaning processes to prove their worth. The best practices endorsed by the consent-decree panel will put an end to that for LSAT takers.
The parties to the original lawsuit have until February 26 to notify the other parties if they plan to challenge any portion of the expert report. So far, no one has issued such a notice. Let’s hope that continues and that LSAC swiftly implements these best practices. This is both an effective way to end a lawsuit and a fair way to treat potential colleagues in the legal profession.
If you would like to offer comments to LSAC, including encouragement for them to accept the recommendations and put this lawsuit behind them, feel free to email LSAC’s Excecutive Director Daniel Bernstine at dbernstine@lsac.org.
I’m proud to say that my colleague, Ruth Colker, was a member of this expert panel. She was chosen by the other four panelists (two apiece from each side of the litigation) from a list of three top-notch experts drawn up by the Department of Justice. In her day job, Ruth serves as a Distinguished University Professor of The Ohio State University and Heck Faust Memorial Chair in Constitutional Law at our College of Law.
About Law School Cafe
Cafe Manager & Co-Moderator
Deborah J. Merritt
Cafe Designer & Co-Moderator
Kyle McEntee
Law School Cafe is a resource for anyone interested in changes in legal education and the legal profession.
Around the Cafe
Subscribe
Categories
Recent Comments
- on Scholarship Advice
- on ExamSoft: New Evidence from NCBE
- on COVID-19 and the Bar Exam
- on Women Law Students: Still Not Equal
- on Ranking Academic Impact
Recent Posts
- The Bot Takes a Bow
- Fundamental Legal Concepts and Principles
- Lay Down the Law
- The Bot Updates the Bar Exam
- GPT-4 Beats the Bar Exam
Monthly Archives
Participate
Have something you think our audience would like to hear about? Interested in writing one or more guest posts? Send an email to the cafe manager at merritt52@gmail.com. We are interested in publishing posts from practitioners, students, faculty, and industry professionals.