The State Bar of California has admitted to using artificial intelligence (AI) to develop a portion of multiple-choice questions on its February 2025 bar exam, leading to furore amongst the faculty members and exam takers. The admission by the state’s legal licensing body comes after the prospective lawyers complained that the exam was plagued with technical problems and irregularities, according to a report in the Los Angeles Times.
On Monday (Apr 21), the State Bar said its psychometrician (a person or organisation skilled in administering psychological tests), ACS Ventures, created 23 of the 171 scored multiple-choice questions with AI assistance. Another 48 questions came from a first-year law student exam, while Kaplan Exam Services developed the remaining 100 questions.
The body added that it will approach the California Supreme Court to adjust test scores for those who took the exam.
“After thoughtful deliberation and hearing directly from many applicants, we believe our recommendations to the Supreme Court mark the beginning of a fair and meaningful path forward for all February bar exam test takers,” said Committee of Bar Examiners (CBE) Chair Alex Chan.
“Our commitment to supporting applicants is unwavering, and we will continue refining our approach to ensure a more responsive, fair, and equitable process now and in the future.”
Also Read | AI Gone Wild: Cursor’s Rogue Bot ‘Hallucinates’ New User Policy
Court advised use of AI?
Mr Chan noted that the court had urged the State Bar to explore “new technologies, such as artificial intelligence” to improve testing reliability and cost-effectiveness.
“The court has given its guidance to consider the use of AI, and that’s exactly what we’re going to do,” Mr Chan added.
However, a spokesperson for the top court in California said that the judges found out only this week that the State Bar had utilised AI to develop exam questions.
Despite the State Bar’s clarification, exam takers and legal education experts have strongly criticised the body for its actions.
“The debacle that was the February 2025 bar exam is worse than we imagined. I’m almost speechless. Having the questions drafted by non-lawyers using artificial intelligence is just unbelievable,” said Mary Basick, assistant dean of academic skills at the University of California, Irvine School of Law.
Ms Basick added that the use of AI to develop questions written by non-legally-trained psychometricians represented “an obvious conflict of interest”.