GPT-4 Passes the Bar Exam

CHICAGO—March 15, 2023—GPT-4, the new multimodal deep learning model from OpenAI, has passed the Uniform Bar Exam, demonstrating an enormous leap for machine learning and proving that an artificial intelligence program can perform complex legal tasks on par with or better than humans, according to a new paper co-authored by Daniel Martin Katz, professor of law at Illinois Institute of Technology’s Chicago-Kent College of Law.

“GPT-4 represents a new frontier in AI’s role in the legal profession and society at large,” says Katz, who collaborated with the legal AI company Casetext and fellow researcher Michael Bommarito. “The bar exam is an interesting test for AI to pass because it highlights the prospect of such technology  becoming a ‘force multiplier’ that expands access to legal services to all members of society, including those who couldn't previously afford to hire a lawyer.”

GPT-4 scored a 75 percent on the bar exam, higher than the 68 percent average and good enough to place in the 90th percentile. In a previous paper that Katz co-wrote, GPT-3.5 scored a 50 percent and passed only two multiple choice portions of the bar exam, placing it in the 10th percentile. 

In this test, GPT-4 not only took the multiple choice sections, but also the essays (worth 30 percent) and performance test (worth 20 percent). Although many have been skeptical about AI’s ability to pass sections that require generating language, GPT-4 did so by a significant margin, giving responses that were generally on par with the “representative good answers” provided by many state bars. The latest GPT model also shows fewer “hallucinations,” in which an AI language model confidently asserts wrong answers that have no basis in reality.

Passing the bar exam requires the command of not just ordinary English, but of complex “legalese,” which is difficult even for humans. GPT’s rapid advancement in this field is sure to have wide implications for the legal profession.

“Lawyers need to figure out how to really use these tools. And those that do, it’ll be a very positive thing for them. We’re sitting on the dawn of a major increase in potential capacity,” says Katz. “These are tools that allow you to more effectively do your work, so you need to learn how to use them to maximum efficacy.”

In a period of roughly four years, the leading large language model family (GPT) has progressed from zero percent on the Multistate Bar Exam for GPT-2 to nearly 76 percent on the Multistate Bar Exam in GPT-4. (The Multistate Bar Exam is the multiple-choice section of the Uniform Bar Exam.) Overall, the authors report a 297 Uniform Bar Exam score for GPT-4, which reflects passing the bar by a fairly comfortable margin. The highest threshold in the country is Arizona at 273 (Illinois is 266). 

Data and code for the bar exam study will soon be made available for viewing.

Photo: This graphic shows the progression of GPT models taking the bar exam.

Illinois Institute of Technology

Based in the global metropolis of Chicago, Illinois Tech was born to liberate the collective power of difference to advance technology and progress for all. It is the only tech-focused university in the city, and it stands at the crossroads of exploration and invention, advancing the future of Chicago and the world. It offers undergraduate and graduate degrees in engineering, computing, architecture, business, design, science and human sciences, and law. Illinois Tech students are guaranteed hands-on experiences, personalized mentorship, and job readiness through the university's one-of-a-kind Elevate program. Its graduates lead the state and much of the nation in economic prosperity. Its faculty and alumni built the Chicago skyline. And every day in the living lab of the city, Illinois Tech fuels breakthroughs that change lives. Visit iit.edu.

Chicago-Kent College of Law

Founded in 1888, Chicago-Kent College of Law is the law school of Illinois Institute of Technology. Located in the third-largest legal market in the country, Chicago-Kent College of Law is positioned on the cutting edge of a rapidly changing legal industry. It offers students an outstanding legal education that combines academic rigor with numerous opportunities for hands-on practice—beginning in year one. The school's trial advocacy program—consistently ranked in the nation's top 10—and award-winning moot court teams allow students to test their rhetorical and courtroom skills against the best competitors in the country. The Chicago-Kent curriculum emphasizes foundational learning and practical skills better than any other law school in Chicago, making it among the best in the country. With its many distinguished J.D. certificate programs, including top-ranked programs in intellectual property and labor and employment law, Chicago-Kent prepares its students not just to succeed, but also to excel in their fields. And as the nation’s only law school tied to a tech university, Chicago-Kent educates graduates who are adept at quickly familiarizing themselves with technological advancements that are constantly altering the practice of law.

Media contact

Kevin Dollear
Communications Manager
Illinois Institute of Technology
Cell: 773-860-5712
kdollear@iit.edu