GPT-4 Beats the Bar Exam

October 26th, 2023 / By Deborah J. Merritt

In the first three posts in this series, I used a bar exam question as an example of the type of problem a new lawyer might confront in practice. I then explored how GPT-4 might help a new lawyer address that practice problem. In this post, I’ll work with another sample question that NCBE has released for the NextGen bar exam. On this question, GPT-4 beats the bar exam. In other words, a new lawyer using GPT-4 would obtain better answers than one who remembered material studied for the bar exam.

Take a look at the seventh multiple-choice question on the NextGen list of sample questions. In this question, the client is a state college that wants to fire a tennis coach. The question format parallels the one we discussed in my previous posts: It asks which “legal topics are the most important for you to research before advising the client” and instructs the test-taker to choose two topics from a list of six.

When prompted with the multiple-choice question, GPT-4 quickly identifies the two correct answers: employment at will and procedural due process. The Bot also offers succinct explanations about why those topics are the most important of the six to research. But that’s not the interesting part–we already know that GPT-4 can easily pass the bar exam.

Instead, I prompted GPT-4 (on a different day and with a new conversation) with the fact pattern in this MCQ and asked: “What legal topics are most important for you to research before advising the client?” Here’s how GPT-4 responded:

---

When representing a state college considering terminating an employee, several legal considerations come into play. Given the provided facts, here are the most important legal topics to research before advising the client:

1. At-Will Employment vs. Contractual Employment:

In many U.S. jurisdictions, unless there is an employment contract stating otherwise, employment is typically considered “at-will.” This means that either the employer or the employee can terminate the employment relationship at any time for any reason (that isn’t illegal). However, if the coach and college had an oral contract, its terms will need to be considered.

2. Oral Contracts:

While written contracts are easier to enforce, oral contracts can still be binding. The challenge is proving their terms and existence. Research would be needed to determine the enforceability of oral contracts in your jurisdiction and what evidence would be required to prove the terms of such an agreement

3. Implied Contract:

Even if there was no formal oral agreement about the contract’s duration, the actions and communications between the college and the coach might have implied a certain term of employment. Research would be needed to determine if an implied contract was potentially formed and its implications.

4. Due Process Rights for Public Employees:

Since the college is a state institution, the coach may have constitutional due process rights. If the coach has a property interest in their job (i.e., a reasonable expectation of continued employment), the state cannot deprive them of this without due process. Depending on local laws and court rulings, this might entail the need for a hearing before termination.

5. Public Policy and Whistleblower Protections:

The coach cannot be terminated for reasons that violate public policy (e.g., reporting illegal activities). Ensure that the budget cuts are the true and only reason for the contemplated termination.

6. Discrimination Laws:

Federal and state laws prohibit employment discrimination based on race, color, religion, sex, national origin, age, disability, and sometimes sexual orientation and gender identity. Ensure the coach’s termination isn’t grounded in any discriminatory reasons.

7. Breach of Good Faith and Fair Dealing:

In some jurisdictions, there’s an implied covenant of good faith and fair dealing in employment relationships. This means that even in at-will employment, employers cannot terminate employees to avoid paying earned benefits or for other bad faith reasons.

8. Potential Damages:

If the coach were to bring a wrongful termination suit and win, what would the potential damages be? This might include lost wages, emotional distress, punitive damages, and attorney’s fees.

9. Alternative Dispute Resolution:

Does the state college have any policies or practices regarding mediation or arbitration before proceeding with termination?

10. Release and Settlement Options:

If the college wants to avoid potential litigation, they might consider offering the coach a severance package in exchange for a release of claims.

After researching these topics, it would be crucial to provide a well-informed opinion to the client about the potential risks and benefits of terminating the coach, as well as any alternative strategies that might minimize legal exposure.

---

This answer includes the two points tested by the MCQ, but it is much more comprehensive. A new lawyer would do well following the leads offered by GPT-4. In contrast, a lawyer who simply remembered material tested on the bar exam would fall woefully short of serving this client competently. Comparing these two responses points out three problems with the bar exam–and with legal education more generally.

First, the subjects tested by the bar exam–and that we teach as “foundational” in law school–do not track contemporary law practice. The discrimination laws cited by GPT-4 are as (or more) critical to an employment law practice as the notion of employment at will. One of the first questions I would ask my client in this situation is whether the coach is a member of any protected class. But discrimination laws are not part of the subject matter tested on the bar exam, so the question doesn’t embrace them.

Second, both law school and the bar exam teach aspiring lawyers to think narrowly. Courses, classroom hours, and exam questions (especially multiple-choice ones) march students through hundreds of isolated doctrinal rules. Except in clinics, we rarely ask students to solve whole legal problems. GPT-4 is absolutely right that the terms of the oral contract, the possibility of an implied contract, and the extent of damages are as important for the lawyer to consider as the concept of employment at will. Yet we rarely encourage students to consider multiple doctrines within a single subject area, much less across subjects.

Students may need to learn some legal principles in isolation but I think we, as legal educators, need to recognize how atomized our teaching and testing have become. Over the last few decades, we have squeezed ever more doctrine into each credit hour, losing opportunities to help students integrate those pieces. Our shift towards multiple-choice testing, meanwhile, reinforces the notion that law consists of discrete legal principles operating in isolation.

Third, our teaching and licensing focus too much on reciting legal rights–and too little on helping clients solve problems more holistically. Even when a client asks only about the “legal repercussions” of an action, as the client did in this bar exam question, an effective lawyer would explore the problem more holistically with the client. Is abrupt termination the only solution to the client’s problem? Could the client offer the coach a severance package that would avoid litigation? Even if the client has no legal obligation to offer the coach a hearing, would a hearing help ease the blow to the coach? Could a hearing even lead to a new solution to the client’s underlying problem (a budget shortfall)? Maybe the coach would be willing to cut back their hours and continue serving the team.

Even without special prompting, GPT-4 recognized one of these more holistic solutions (a severance package) as an issue for the new attorney to consider. The bar exam question, in contrast, forces the aspiring lawyer to focus single-mindedly on the “legal topics.” Who is the machine in this comparison? Could working with GPT-4, ironically, lead us back to more holistic problem-solving with our clients?

AI, Teaching, Bar Exam, GPT-4