Dean Le BlancCollege faculty member. Still grades every week.
The most useful thing about the legal AI that shipped this week is not how good it is. It is that its maker told you how often it is wrong.
On Thursday OpenAI introduced Astra for Law: its most capable model wired to a licensed index covering more than 230 million URLs and over 99.9 percent of published United States precedential case law. Serious engineering, with named firms building on it.
Then, in the same announcement, they published the score. On 200 questions from Vals AI’s Legal Research Bench, at the highest reasoning effort, it passed the overall correctness check on 54.0 percent. The same model using ordinary web search passed 38.7.
Read the second number before you react to the first. That jump is a forty percent relative improvement, and it is what a licensed corpus buys. It also leaves forty six percent not passing. Both are true, and holding both at once is the whole skill.
Law now knows its number, its benchmark and its baseline. Your discipline knows none of that, and nobody is preparing to tell you. The best measured profession will be the one with the most published failures, because publishing a failure rate is what a serious vendor does. Silence is what everyone else does.
Week six, and the first real papers are landing. If your term started in mid August you are now reading the first submissions that went through a full AI cycle: draft, polish, submit. This is the week the AI policy you wrote in July meets the work.
Check what your policy actually names. Mine named ChatGPT for two terms. That policy did not cover Codex. It did not cover a firm or campus interface running the same model under a different label. It will not cover whatever ships in March. Same system, different name on the screen, policy silent.
Midterm grade submission is close for most terms. Anything you want to change about how AI use is documented has to change before the next assignment goes out, not after the grade dispute.
Astra for Law. And you cannot have it.
This is the honest part, and it is the part the coverage will skip. Astra for Law is going to selected law firms through a Trusted Access program in ChatGPT and Codex. OpenAI says API availability is coming soon rather than today. There is no higher education tier announced, no student tier, no library licence, and no path for a law school clinic that is not already an eligible firm.
So if you teach in a law programme, the realistic position this term is that your students will read about a tool their future employers are piloting and will not be able to touch it, while the general model they can touch is the one that scored 38.7.
Is it worth your attention anyway? Yes, for one reason, and it is not the capability. It is that OpenAI published a number against an independent benchmark and named the baseline it beat. That is a norm worth rewarding, because the moment a vendor starts quoting capability without a denominator you will have something to compare it against.
Do not bother trying to get access, trying to build a course around it, or putting it on a spring syllabus. It is not procurable by you. Read the announcement, take the number, and spend your energy on the part you control.
Look closely at OpenAI’s own published comparison and there is a detail that should stop a faculty member cold. In their side by side example, a competing frontier model returned a holding that had been reversed on appeal.
Not a hallucinated case. A real case, correctly cited, wrong in the way that matters most: no longer good law.
Every AI detection conversation in higher education is still organised around the question of whether a student wrote it. That failure mode is about whether the thing is still true. A detector cannot see it. A plagiarism check cannot see it. A citation checker that confirms the source exists cannot see it either, because it does exist.
The only thing that catches it is a human opening the source and reading far enough to find out what happened to it. That is a teachable, gradeable, thirty second habit, and almost no assignment currently asks for it.
Write one verification clause, for one assignment, by hand. Fifteen minutes.
Take the next assignment you have due. Not the hardest one. On paper, answer four questions.
One. Which classes of claim have to be checkable in this assignment? Not “all sources.” Name the two or three that actually carry risk here. Cited cases. Figures pulled from a table. Attributed quotes. Dates.
Two. What is the check, as an action ending at a primary source? Start with a verb. “Open the case and confirm it has not been reversed.” “Pull the figure from the original report, not a summary of it.”
Three. What does the student hand in that proves the check happened? A source line with the date checked. A page number. One sentence.
Four. How does a student report a claim they could not verify? Write the actual sentence they are allowed to use, and say what it is worth. This is the part almost every AI policy leaves out, and it is the part that changes behaviour, because if there is no way to say “I could not confirm this one,” the only move available is to hide it.
That is your clause. It names the step instead of the software, which is exactly why it survives a rename, a new vendor and a model upgrade. It would also have caught the reversed holding.
Now look at what you just wrote, and count the assignments in that one course, and then count your sections. The clause is not the hard part. Writing it eleven more times, in the same voice, consistently enough that students stop reading it as boilerplate, in every course you teach, is the part that quietly does not happen by week six. That specific job is what the Syllabus Generator does, off your Course Record, so a clause you wrote once carries into every assignment and every section.
And do the first one by hand anyway. A clause you wrote yourself is the one you will actually enforce.
If you would rather be walked through it, the session is here and it is free: The Verification Clause.
- Monday: Free AI Curriculum Has a Byline. Read It Before Your Department Votes. The four question audit for donated curriculum.
- Tuesday: the first grading weekend, and the rubric you are about to regret.
- Wednesday: A Court Fined the Signature, Not the Chatbot. Your Assignment Has the Same Gap.
- Thursday: The Blank Page Breaker, a working session for students staring at a paper due Sunday.
- Friday: today’s post, and The Verification Clause.
Law found out its number this week because somebody chose to publish it.
The next vendor who walks into your department meeting will have a number too. The only question is whether anybody in the room asks for it, and what you do in the silence after you do.
Forward this to the colleague who is writing the spring syllabus.
Harvest Kernel for the long teaching sessions. Tuesdays faculty, Thursdays students.
The Daily Seed for the short ones.