AI Grading Makes a Harsh Mark Harder to Fix. Grade Blind First.

Checking an AI grade feels like oversight. In a 1,339 teacher experiment, a grade that was three points too harsh survived review more often when the label said AI.

That experiment sits at the center of a sharp piece on AI grading that Jim Dickinson, an associate editor at Wonkhe, published on October 13 under the headline “AI marking could cost universities their get out of jail free card.” His argument is written for universities in the UK. The mechanism underneath it is not British. It is about the order in which a person and a machine look at the same piece of student work, and it applies to anyone who grades.

Most conversations about AI grading stop at one reassurance: a human stays in the loop. The model takes a first pass, the instructor checks it and signs it off. Dickinson’s point is that the order of those two steps decides whose judgment the grade actually carries, and that is the question most AI grading policies skip.

What 1,339 teachers did with a wrong grade

The study was published this year in PNAS Nexus by Sofoklis Goulas, Rigissa Megalokonomou and Panagiotis Sotirakopoulos, and it was preregistered and randomized. Teachers across Greece each saw the same piece of student work with a grade of 5 out of 10 already attached, plus a list showing which of its five answers were right and which were wrong, so the fair grade could be worked out. Some were told the 5 came from a colleague. Others were told it came from an algorithmic grading system.

In one version the fair grade was 8, so the 5 was three points too harsh. In the other the fair grade was 2, so the 5 was three points too generous.

When the 5 was too generous, the label made no meaningful difference. Teachers corrected it about as readily whether it came from a colleague or from AI. When the 5 was too harsh, the label mattered. The distance between the teacher’s final grade and the fair grade was 22 percent larger when the harsh grade was labeled as coming from AI. In the authors’ own words, teachers allowed a harsh AI score to stand notably more often than the identical human score.

Dickinson’s summary is the line worth keeping: the deference runs in the direction that hurts students.

Be fair to the evidence before you apply it to AI grading in your own course. These were school teachers working through a short exercise, not professors reading a term paper at midnight. But the mechanism is one every grader knows. Once a number is on the screen, it becomes the default, and disagreeing with it becomes work.

The regulator saw the same thing in AI grading workflows

In January, Ofqual, the regulator for qualifications such as GCSEs and A levels in England, published a working paper on AI in marking. It compares several ways a person and a model might share the job.

For the arrangement most universities would call human in the loop, where AI generates the marks and a person reviews every one, the paper lists two problems. The first is no efficiency gain. The second is an “anchoring effect and/or risk of uncritical human review.” For the arrangement where AI marks in parallel to the human marker and only the disagreements get flagged for review, its note is two words: “Avoids anchoring.”

The same paper is plain that using AI as the sole mechanism for determining a student’s mark does not comply with Ofqual’s regulations.

Read those together and the trade is uncomfortable. If you review every AI mark properly, you redo the thinking and save nothing. If you do not redo the thinking, the judgment is no longer yours.

Who, or what, decided this mark?

This is where Dickinson’s headline comes from. Under the UK’s Higher Education Act 2004, a complaint to the Office of the Independent Adjudicator, the body that reviews student complaints against universities, does not qualify “to the extent that it relates to matters of academic judgment.”

The adjudicator’s own guidance defines that narrowly: “Academic judgment is not any judgment made by an academic; it is a judgment that is made about a matter where the opinion of an academic expert is essential.”

So the protection rests on an expert having looked at the work and formed a view. A student cannot argue with the view. A student can ask whether a view was ever formed. As Dickinson puts it, “who, or what, decided this mark?” is a question of fact.

The adjudicator has already seen an early version of this. In a case summary from July 2025, an MBA student argued that their dissertation feedback was partly AI generated, pointing to a detection tool. The university said AI had not been used. The adjudicator decided it was reasonable to give more weight to the staff member’s account of how they had marked the work than to the detector. Dickinson’s sequel is the uncomfortable part: once AI is formally part of the marking workflow, the marker’s account changes.

You do not need to teach in the UK for this to land. We made a related argument when Dartmouth authorized an AI detector and its own rules said the score is not the case, and again when Alaska’s draft AI policy said a human decides. Open your own institution’s grade appeal policy and find the sentence that says what an appeal can and cannot question. Then ask whether you could say, with a straight face, that every grade you returned this term came from your own expert view.

Same essay, different grade

There is a second problem, and it undercuts the strongest argument for AI grading, which is consistency.

In September 2025, researchers at Thinking Machines Lab sent the same prompt to an open model 1,000 times at temperature zero, the setting meant to make output repeatable. They got 80 different completions. They traced the cause to the way the server batches requests, which shifts with load. In plain terms, the answer could depend on how busy the server was.

Human graders disagree too. But two experts disagreeing is a difference of professional opinion. The same tool giving the same essay two different marks on two different afternoons is something else, and as Dickinson notes, a student can run the same essay through the same tool ten times and screenshot the spread.

Mark first. Then let the machine look.

None of this says AI grading has no place in your course. It says the order matters, and the order is yours to set. Dickinson sketches one version, and it lines up with the design Ofqual says avoids anchoring.

One. Mark blind. Read the work and record your mark and your reason before any tool shows you a number. One line quoted from the student’s work, and one sentence on why it earned the mark.

Two. Let the tool look second. If you use a tool, let it mark in parallel, and compare only after your mark exists.

Three. Set a gap rule before you start. Decide in advance how far apart your mark and the tool’s can be before you take a second look, and write down what you decided and why.

Four. Keep the record. If a tool was involved, note which one, which version and the date. Dickinson points out that a mark generated on a model that has since been retired cannot be regenerated, so the record is the only account you will have.

Five. Tell students. The University of Newcastle in Australia already requires that any use of generative AI in marking be communicated to students, with students given the opportunity to opt out.

Where AI carries less of this risk is after the judgment exists, turning your notes into a comment a student can actually use. Dickinson notes that some UK universities draw exactly that line in their guidance: AI can tidy an academic’s comments, but it cannot originate the judgment behind them.

One essay we keep coming back to, Solve Everything, argues for decision records for AI systems, a flight recorder for why a high stakes automated decision came out the way it did. A grade is a high stakes decision. Your judgment record is the flight recorder, and it only works if you write it before you look.

Your next step

Take one paper from the stack you are grading this week through The Blind First Mark, by hand, in about 20 minutes.

When you finish you will hold a judgment record for one criterion: your mark, the line from the student’s work that decided it, your reason, the tool’s mark if you use one, your gap rule, and one sentence for students about how AI is used in your grading. Then the next thought arrives on its own. That is one paper. The rest of the stack is waiting, and every one of them deserves the same order, your judgment first.

The judgment cannot be handed off. The typing can. Turning each mark into feedback a student can use, rubric aware and in your tone, is the job the Student Feedback Generator does. It grounds itself in your Course Record, the five minute record of your course, your outcomes, your policies and your voice that every tool in the toolkit uses. Bring your judgment first, then let it draft the comment.

If you would rather think it through with other instructors first, the Harvest Kernel Learning Community is free, and this question is the conversation there this week.

Sources

• Jim Dickinson, “AI marking could cost universities their get out of jail free card,” Wonkhe, October 13, 2026. wonkhe.com
• Sofoklis Goulas, Rigissa Megalokonomou and Panagiotis Sotirakopoulos, “Why do experts miss AI’s errors? Evidence from a randomized labeling experiment,” PNAS Nexus, 2026. doi.org
• Ofqual, “Principles of AI use in marking,” working paper, published January 14, 2026. gov.uk
• Higher Education Act 2004, section 12. legislation.gov.uk
• Office of the Independent Adjudicator, “Complaints we can and cannot review (Rule 4 and Rule 5).” oiahe.org.uk
• Office of the Independent Adjudicator, “AI and academic misconduct, CS072506,” case summary, July 2025. oiahe.org.uk
• Horace He and Thinking Machines Lab, “Defeating Nondeterminism in LLM Inference,” September 10, 2025. thinkingmachines.ai
• The University of Newcastle, Australia, “Policy on the use of Generative AI in Teaching, Learning and Assessment.” policies.newcastle.edu.au
• “Solve Everything.” solveeverything.org