AI Didn’t Only Pass the Bar Exam — It Assisted in Creating It

Artificial intelligence has already demonstrated its ability to pass the bar exam, but now it is taking on a new function: assisting in the formulation of the test itself.

This week, the State Bar of California revealed that AI was utilized to generate a segment of the questions on its latest bar exam, as reported by the Los Angeles Times. The questions crafted by AI were produced by ACS Ventures, an independent psychometric consulting firm that the State Bar hired. These questions were generated with AI support and subsequently evaluated by subject matter experts and validation panels prior to being incorporated into the exam.

In total, 23 out of 171 multiple-choice questions on the exam were devised by ACS Ventures with AI tools. Most of the questions were authored by Kaplan, while a few were taken from the First-Year Law Students’ Exam. This year’s bar exam was conducted remotely for California examinees — a decision that had already caused frustration due to widespread technical difficulties and system failures.

The disclosure that AI participated in formulating some of the exam questions has further fueled criticism from students and legal educators alike.

“I can hardly find the words,” remarked Mary Basick, assistant dean of academic skills at UC Irvine School of Law, in an interview with the LA Times. “It’s just unbelievable that non-lawyers using artificial intelligence drafted the questions.”

Katie Moran, an associate professor at the University of San Francisco School of Law, shared this concern, describing it as “a staggering admission.” She also pointed to a possible conflict of interest, mentioning that ACS Ventures not only generated the AI-produced questions but also validated them for use.

Alex Chan, chair of the State Bar’s Committee of Bar Examiners, informed the Times that the California Supreme Court had urged the Bar to investigate emerging technologies like AI to enhance the exam’s efficiency and dependability.

The employment of AI in high-stakes scenarios such as legal testing reflects a broader movement. With the emergence of generative AI tools like ChatGPT, automation has progressed beyond mundane tasks to include work with serious real-world consequences. For example, some believe that AI might have been utilized to calculate tariff rates during the Trump administration. In another instance, two New York attorneys faced sanctions in 2023 for submitting a legal brief citing fictional cases produced by ChatGPT. Academic journals have also experienced a rise in publications featuring AI-generated material.

These occurrences underscore the increasing dependence on AI for intricate tasks — along with the risks associated with it. Generative AI can create convincing yet inaccurate information, a phenomenon termed “hallucination.” In the realm of the bar exam, where a single test can influence a law student’s career trajectory, the ethical ramifications of employing AI are especially profound.

As AI continues to transform professional landscapes, the discussion regarding its role in essential processes like legal certification remains ongoing.