OpenAI Built a Model Just for Lawyers. Its Own Benchmark Says 54 Percent.

Fifty four percent. That is OpenAI’s own number, for OpenAI’s own new legal model, on the benchmark OpenAI chose.

On September 17 OpenAI introduced Astra for Law, and it is not a wrapper or a prompt pack. It is the company’s most capable model wired to a purpose built legal search index covering United States case law, statutes, regulations, court rules and administrative decisions across more than 230 million URLs. Through a collaboration with Free Law Project, the nonprofit behind CourtListener, it reaches more than 99.9 percent of published United States precedential case law. Sullivan and Cromwell, Ropes and Gray, and Cooley have all built working tools on it. Twenty six partner plugins shipped alongside it, plus nine community plugins carrying forty seven custom skills.

This is as serious as domain specific AI gets. A profession, a licensed corpus, named firms, published evaluations.

And on 200 legal research questions from the private validation set of Vals AI’s Legal Research Bench, at the highest reasoning effort, Astra for Law passed the overall correctness check on 54.0 percent of them. The same model using ordinary web search passed 38.7 percent.

The number is not the story. The fact that there is a number is the story.

Read that jump again, because the temptation is to read it as bad news and it is not. Going from 38.7 to 54.0 is a forty percent relative improvement, bought with a licensed index and instructions written by people who actually practice. That is what real domain engineering buys, and it is a lot.

It also leaves forty six percent of those questions not passing the correctness check. Both of those sentences are true at once, and a faculty member who can hold both of them is already doing something most AI policy documents cannot do.

Here is the part that matters for your course. Law now knows its number. It knows the benchmark, the question count, the reasoning setting, and the baseline it was measured against. Your discipline does not know any of that, and nobody is preparing to tell you.

Before you go further: the working session at the end of this post gets you one finished verification clause for one real assignment, written in your words, in about fifteen minutes. Open The Verification Clause and do it alongside the reading.

What a published number actually changes

There is a line in Dean’s own notes, saved long before this launch, that gets closer to the problem than most of the coverage will: large language models are confident liars when they are unguided. The fix in those notes was never to find a model that does not err. It was to force the model to separate fact from inference, cite sources with dates, and rate its own confidence, so the human reads an audit instead of a verdict.

That is what a published benchmark does at the institutional scale. It converts a vibe into a number a department can plan around. You cannot write a sensible policy against “AI is sometimes wrong.” You can write a very sensible policy against “this tool passes fifty four percent of research questions in its own maker’s evaluation, so every output carries a verification step before it leaves the building.”

Notice what happens to the conversation once the number exists. The argument stops being about whether students and professionals should use the tool. It becomes a question about what has to happen between the output and the submission. That is a much better argument, and it is the one your syllabus is currently not having.

Your policy names a product. Products get renamed.

Astra for Law is going to selected firms first through a Trusted Access program in ChatGPT and Codex. OpenAI says it will appear in the model picker as GPT 6 Astra Law, and that API availability is coming soon rather than today.

Sit with the naming for a second. A student or a junior associate in a professional program this term may be working in ChatGPT, in Codex, in a firm’s own interface built on the same model, or in a partner product like Harvey or Legora. Same underlying system. Four different names on the screen.

Now open your syllabus and read your AI policy out loud. If it names ChatGPT, it does not cover any of the other three. If it names “generative AI tools,” it covers everything and requires nothing. Both versions fail for the same reason: they describe the software instead of describing the work.

This is not a hypothetical drift. It is the third time this term that a tool your policy names has changed shape underneath it, and the pattern is not going to slow down while institutions catch up.

Name the step, not the software

The move is to stop writing policy about tools and start writing policy about verification. A clause that names a step survives a rename, a new vendor, a model upgrade and a student who found something you have never heard of.

A verification clause has four parts, and none of them require you to know which model anybody used.

1. The claim class. Which kinds of statement in this assignment have to be checkable. Citations, statutes, dates, figures, attributed quotes. Name them specifically for this assignment rather than in general.
2. The check. What the student does to verify each one, stated as an action against a primary source. Open the case. Read the holding. Confirm it has not been reversed.
3. The record. What they hand in that proves the check happened. A source line with a date, a screenshot, a short note on what they could not confirm.
4. The honest gap. Credit for saying “I could not verify this one” instead of hiding it. This is the part everyone leaves out, and it is the part that changes student behavior, because an unverifiable claim becomes something to report rather than something to bury.

Read that against the OpenAI announcement one more time. In their own published comparison, a rival model returned a holding that had been reversed on appeal. That is not an exotic failure. That is exactly what part two and part three of the clause above are built to catch, and a student running that check finds it in ninety seconds.

Notice that a clause like this is also a teaching instrument rather than a compliance instrument. It is the Grow phase of SeedStacking doing what it does: the student is not being policed, they are being handed the professional habit that the profession itself is now being sold as a feature.

The professions are going first, and higher education is downstream

Law is not special here. It is early. It has a licensed, machine readable corpus, wealthy clients, and malpractice exposure, which is exactly the combination that gets a vertical model built first. Medicine, accounting, engineering and architecture have the same ingredients in different proportions.

So the honest forecast is not that a legal AI arrived. It is that the disciplines you teach are getting purpose built models on a schedule set by procurement departments, not by curriculum committees, and each one will arrive with a name your syllabus does not contain.

The Course Record exists for exactly this reason. When your outcomes, your policies and your voice live in one place, a change in the tool landscape is an edit rather than a rewrite. When they live in eleven separate syllabus documents, it is a semester of work nobody has.

Do one, by hand, today

Take one assignment. The next one you have due, not the hardest one. Write the four part verification clause for that single assignment in your own language. Fifteen minutes, on paper, no tool required. The working session below walks you through it and hands you the finished clause.

Then look at what you just wrote, and look at how many assignments are in the course, and how many sections you teach. That is the moment worth paying attention to. The clause is not hard. Doing it eleven times, consistently, in a voice that matches the rest of your syllabus, in every course you teach, is the part that does not happen. That is the specific job the Syllabus Generator does, and it is worth doing the first one by hand anyway, because a clause you wrote yourself is the one you will actually enforce.

Law found out its number this week because somebody published it. Nobody is going to publish yours.

Sources

• OpenAI, Introducing Astra for Law, September 17, 2026. All figures in this post, the 54.0 percent and 38.7 percent benchmark results, the 200 question validation set, the 230 million URL corpus, the 99.9 percent case law coverage, the plugin counts and the model naming, are taken from this announcement.
• Free Law Project, CourtListener case law data coverage.