Comparisons

April 29th, 2015 / By Deborah J. Merritt

Michael Simkovic has posted some comments on my study of recent law graduates in Ohio. I had already benefited from his private comments, made some changes to my paper, and thanked him privately. When he reads the revised paper, posted several weeks ago, he’ll discover that I also thank him in the acknowledgement footnote–with the disclaimer that he and my other readers “sometimes disagree with one another, as well as with me, which made their comments even more helpful.”

For those who are interested, I note here my responses to the critiques that Professor Simkovic offers in his blogpost. Beyond these comments, I think readers can judge for themselves how much my study helps them understand the market for law school graduates in their part of the world. Some will find it relevant; others will not. As I’ve already noted, I hope that others will collect additional data to complement these findings.

Here and There

Professor Simkovic’s primary criticism is that the Ohio legal market is not representative. I discussed that issue in a previous post, so will add just a few thoughts. It is true that the wages for Ohio lawyers fall below the national average (both mean and median), but Ohio’s cost of living is also below average. Our index is 94.1 compared to 128.7 in California, 133.3 in New York, and 141.6 in the District of Columbia. On balance, I don’t see any reason to dismiss Ohio as a representative state for this reason.

Lawyers constitute a smaller percentage of the Ohio workforce than of the national one, but that is not a particularly meaningful indicator. Oklahoma, with 4.48 lawyers per 1,000 jobs, comes very close to the national average of 4.465, but that would not make Oklahoma the best choice for a study of new lawyers’ employment patterns.

Ohio has a disproportionate number of schools that rank low in the US News rankings: We have one school in the top 50, two in the second tier, three in the third tier, and four among the unranked schools. I discuss the implications of this in my study and show how law school rank affects employment patterns within the state. Like other studies, I find strong associations between law school rank and job type.

It is hard to know how this issue affects my overall findings. Professor Simkovic suggests that low-ranked schools create a sub-par job market with depressed outcomes for graduates. Just the opposite, however, could be true. Ohio individuals and businesses have the same legal needs as those in other states and, as noted above, we do not have as many lawyers per worker as some states. It is possible, therefore, that graduates of low-ranked schools have better employment opportunities than graduates of similar schools in other states. Similarly, the graduates of our first- and second-tier schools may fare better than graduates of similar schools in states with local T14 competitors.

The results of my Ohio study undoubtedly generalize better to some markets than to others. Similarly, the results may interest educators at some schools but not others. I doubt that my study will influence decisions at top-twenty law schools. At other schools, however, I think professors and deans should at least reflect upon the findings. Most of us graduated from elite law schools 10, 20, 30, or even 40 years ago. Our impressions of the employment market were molded by those experiences, and it is very hard to overcome that anchoring bias. I hope that my results at least provoke thought and further research.

Now and Then

Professor Simkovic and others also criticize my attempt to compare the 2014 Ohio data with national data gathered by NALP and the After the JD (AJD) study. I agree that those are far from perfect comparisons, and I note the limits in the paper. Unfortunately, we don’t have perfect data about employment patterns in the legal profession. In fact, we have surprisingly little data given the importance of our profession.

Some of the data we do have is out-of-date or badly skewed. Professor Simkovic and others, for example, cite results from the AJD study. That study tracks the Class of 2000, a group of graduates with experiences that almost certainly differ from those of more recent graduates. The Class of 2000’s history of debt repayment, for example, almost certainly will differ from that of the Class of 2010. In 2000, the average resident tuition at public law schools was $7,790–or $9,864 in 2010 dollars. By 2010, tuition had more than doubled to $20,238.

Rather than rely on outdated information, my study begins the process of providing more current data. (I don’t study tuition in particular; I note that example because Professor Simkovic uses AJD for that purpose in his post.) In providing that information, I also make comparisons to the baseline data we have. Although the prior data stem from different populations and use somewhat different methods, some of the differences are so large that they seem likely to reflect real changes rather than methodological artifacts.

AJD, for example, found that 62.1% of the class of 2000 worked in law firms three years after graduation. At a similar point (4.5 years post graduation), just 40.5% of my population held positions in firms. Some of that difference could stem from method. AJD relied upon survey responses, and the responses showed some bias toward graduates of highly ranked schools. AJD also examined a national sample of lawyers, while I looked only at Ohio. A national sample, however, is not a New York or California sample. AJD included lawyers from Tennessee, Oklahoma, Utah, and Oregon, as well as some from the larger markets. Ohio will not precisely mirror those averages, but I doubt the difference is large enough to account for the 20-point drop in law firm employment.

Assumptions About Non-Respondents

In my study, I tracked employment outcomes for all 1,214 new lawyers who were admitted to the Ohio bar after passing one of the 2010 exams. Using internet sources I was able to confirm a current (as of December 2014) job for 93.7% of the population members. For another 1.6% I found affirmative indications that the population member was not working. I.e., the person had noted online that s/he was jobseeking or that s/he had decided to leave the workforce to care for a family member.

That left 4.7% of the population for which I lacked information. For the reasons discussed on pp. 15-17 of the paper, I elected to treat this group as “not working.” There are some licensed lawyers who hold jobs without leaving any internet trace, but it’s a difficult task. For starters, Ohio’s Supreme Court requires all bar members to notify the court of their current office address and phone; the court then publishes that information online.

In addition, most working lawyers want to be found on the internet. With employer websites, LinkedIn, and Google searches, I found most of the population members very easily. The ones I couldn’t find became intriguing challenges; I returned to them repeatedly to see if I could find any traces of employment. The lack of any such evidence, combined with the factors cited in my paper, suggested that these individuals were not working.

It is quite possible, of course, that some of these individuals held jobs. Any bias toward understating employment outcomes, however, was likely outweighed by countervailing biases: (1) Some online references to jobs persist after an employee has left the position and is seeking other work. (2) My data collection could not distinguish part-time and full-time work, so I gave all jobs the same weight. (3) Some job titles may be polite masks for unemployment. A “solo practitioner,” for example, may not be actively handling cases or seeking clients. (4) My study included only law graduates who were admitted to the bar; it does not include the 10-12% of graduates who never take or pass the bar.

As I acknowledge in the paper, all of these biases could lead to overstating employment outcomes.

Salaries Within Job Categories

Professor Simkovic notes that my study does not account for salary increases within job categories. As I note in the paper, I gathered no data about salaries. I certainly hope that 2010 graduates received salary increases during the last five years! That, however, is a different question from whether employment patterns have shifted among new attorneys. Within the population I studied, I observed several features that differ from employment patterns reported in earlier studies of lawyers. These include the emergence of staff attorneys at BigLaw firms, a notable percentage of solo practitioners, a surprisingly low percentage of lawyers employed at law firms, and substantial percentage of recently licensed lawyers working in jobs that do not require bar admission.

Selection Bias

Professor Simkovic suggests that my study suffers from selection bias because the most talented Ohio graduates may have moved to other states to accept BigLaw offers. This would be a concern if I were trying to describe employment opportunities for a particular law school, but I am not doing that. Instead, I analyze the employment opportunities within a defined market. One can debate, as we have, how well Ohio represents outcomes in other markets. The study, however, is relatively free of selection bias within its defined population. Unlike AJD and many other studies, it does not depend upon subjects’ willingness to answer a lengthy survey.

For the record I’ll note that, although some of my school’s graduates move to other states for BigLaw jobs, the number is small. Like most law schools outside the top-ranked group, we place relatively few graduates at schools with more than 500 (or even more than 250) lawyers. My relatively informed, yet still anecdotal, impression is that our students who move out of state show a similar job distribution to those who remain in Ohio.

What Do We Know?

From my study, we know some things about the jobs held by lawyers who passed the Ohio bar exam in 2010. We don’t know about lawyers who passed the Ohio bar in other years, or about law graduates living in Ohio who have not been admitted to the bar. Nor do we know anything with certitude about lawyers in other states or at different times. But do the facts we know about one set of lawyers at one time provide insights into the experiences of other lawyers? Much social science research assumes that such insights are possible. The reach of those insights depends on the nature of the study.

Here, I think we gain some insight into employment patterns for recent graduates from many schools–at least for the 90% of schools ranked outside the US News top twenty. Some schools and some markets are very distinctive, but most of us are not as different as we first believe. Our first-hand impressions of our graduates’ job outcomes, meanwhile, are very skewed. After just a few years of teaching, we all have lots of former students. The ones we hear from or see at reunions almost certainly differ from those who drop out of sight. Research about Ohio won’t tell you everything you want to know about another market, but it may tell you more than you think.

Can we also gain insights about whether the job market for new lawyers has changed? That is a central claim of my study, buttressed by comparisons to previous data as well as information about why outcomes may have changed. Once again, I think the comparisons add to our knowledge. Personally, I don’t find the fact of change surprising. The legal employment market was different in the 1980s than in the 1950s, and both of those markets were different from the 1920s or 1890s. Why would we in 2015 be exempt from change?

The fact that change has occurred doesn’t mean that the demand for lawyers has evaporated; Richard Suskind’s provocative book title (The End of Lawyers?) has skewed discussions about change by creating a straw man. In the end, even Susskind doesn’t believe that lawyers are doomed to extinction. I think it’s important to know, however, that changes are occurring in the nature of legal employment. Staff attorneys, contract workers, and legal process outsourcers play a larger role today than they did ten years ago; an increasing number of new lawyers seem to establish solo practices; and junior positions in law firms seem to be declining. These and other changes are the ones I discuss in my paper. I hope that others will continue the exploration.