AI-Enabled: Law Professor Highlights Potential of Language Processing Tools in Legal Realm

When OpenAI released its chatbot, ChatGPT, in November 2022, it seemed to take over the internet. Every day someone was demonstrating what the language processing tool was capable of—including almost passing the bar exam.

Chicago-Kent College of Law Professor Daniel Martin Katz, along with a colleague, proved that OpenAI’s GPT-3.5 software is almost capable of passing the bar exam. ChatGPT is the popular free demo version of a fuller program, TEXT-DAVINCI-003, better known as GPT-3.5.

In his paper, "GPT Takes the Bar Exam,” Katz and co-author Michael J Bommarito III outline the process for teaching GPT-3.5 not just to take the Multistate Bar Exam (MBE), but for it to pass two portions—evidence and torts. He says a full pass is right around the corner.

“We say in the paper that in 0–18 months it will pass the bar exam, I’ll stick by that,” he says.

But passing the bar exam wasn’t the point of the research. Katz says this software isn’t going anywhere, and he wants to show that lawyers have to get on board or risk falling behind.

“When people see stuff like this and think that they’re going to continue the way they work, with no changes, I think that’s a very unrealistic perspective,” he says.

OpenAI isn’t the only company developing artificial intelligence systems. Katz says he expects to see multiple language processing models available commercially by the end of 2023, and they’ll only get more complex.

“Three years ago, GPT-2 could not even process the question prompts effectively,” he says.

Now, GPT-3.5 answered 50.3 percent of the questions on the MBE right. Humans get about 68 percent of the answers correct.

“To pass the test, you have to have a pretty good command of not just the English language, but the nuances of ‘legalese,’ or legal language, and that’s a difficult task, even for humans,” Katz says. “You have to go to seven years of postsecondary education, just to get to a test that one in five people don’t pass.”

Now that these technologies are improving so rapidly, Katz says chatbots will likely change the way that people live their lives and do their jobs sooner than many of us expect.

That includes those in the law profession.

“Lawyers need to figure out how to use these tools. And those that do, it’ll be a very lucrative thing for them. We’re sitting on the dawn of a major increase in potential capacity,” says Katz. “These are tools that allow you to more effectively do your work, so you need to learn how to use them to maximum efficacy.”

Change can be intimidating, and a new technology can sometimes translate to lost jobs, though Katz doesn’t think this is the case.

Even though the tools can help, they’re not a replacement for a licensed attorney. GPT can still “hallucinate,” or give answers that sound intelligent, but are actually nonsense. There are bugs, and Katz says there needs to be a “human in the loop.”

“A person using tools like this is able to do a lot more, but they’re the ones providing the oversight,” he says. “If your lawyer uses a research tool like Westlaw or LexisNexis, they still have to stand behind the use of the tool. There needs to be a human in the loop. You shouldn’t use these tools with no human oversight, but a person using this has a jetpack now.”

Beyond improving efficiency for lawyers, these AI tools could have major ramifications for people in need who may not be able to access the legal system due to financial constraints.

“There’s a lot of unmet legal need in America, in the world. We have people who will pay for that need, but it has to be price-appropriate,” says Katz. “Lawyers are simply too expensive, and part of that problem, in my view, is that they don’t have a force multiplier.”

Katz believes that AI could be that force multiplier that the legal realm has needed for years. Many law offices are already using AI to generate contracts and motions.

These tools could even help people who can’t afford a lawyer to assist them in finding some of their own answers.

“A lot of people out there are using Google as their de facto lawyer,” he says. “Let’s try to make sure there’s some high-quality legal information available on the internet, so they can have at least some reasonably trustworthy sources that they can rely on.”

Now that Katz has proven the impact that GPT can have in the legal community, there are likely conversations happening in law schools and offices across the country about what to do next.

“One day soon, lawyers will be expected to know how to use these tools,” Katz says. “And schools will have to decide what they want to do about that. Because these tools aren’t going away.”

Photo: Professor of Law Daniel Martin Katz