Harvest Kernel · Interactive Tool

The Evidence Test

A 4-question filter for any AI education tool, vendor pitch, or “this changed everything” testimonial. Run the test before you adopt.

This is the test we run on every AI education tool before we let it near a student, a curriculum, or a budget. It is borrowed from clinical practice: the bigger the claim, the stronger the evidence needs to be.

Most tools fail at least one question. The ones worth your time pass all four.

What are you evaluating? (optional)

Question 1

What exactly is the claim?

A real claim names the outcome, the population, and the time horizon. “Improves learning” is a vibe. “Students score 18% higher on transfer tasks two weeks after instruction” is a claim.

Specific. Names a measurable outcome, who it applies to, and over what time frame.
Partial. Names an outcome but not the population or time horizon (or vice versa).
Vague. Mostly marketing words: “engaging,” “personalized,” “transformative,” “empowers.”
None. I cannot rewrite their claim as a single measurable sentence.

Question 2

Does the evidence match the size of the claim?

Bold claims (“replaces a tutor”) need controlled studies, comparable groups, replication. Modest claims (“students engage more”) only need modest evidence.

Strong match. Controlled study, comparable groups, replicated at multiple institutions.
Reasonable. Pilot data with controls or a peer-reviewed study, even if not replicated.
Weak. Vendor pilots, glowing case studies, or self-reported metrics only.
None. Testimonials, marketing language, or “trust us” only.

Question 3

Does it build a skill or hide one?

A skill-building tool makes the learner measurably better when the tool is OFF. A hiding tool only works while it is on. Same output, opposite long-term effect.

Builds. Evidence the learner performs better on related tasks without the tool, weeks later.
Probably builds. Designed to fade out support (scaffolding) but no transfer data yet.
Unclear. Output looks better with the tool on, no measurement of what happens without it.
Hides. Tool does the work the learner is supposed to be doing. Performance drops when off.

Question 4

Who is the evidence FROM, and who is it FOR?

A vendor case study is marketing. An independent peer-reviewed study is evidence. Vendors, administrators, and students often want three different things from the same tool.

Independent. Peer-reviewed, independent funding, replicated by people without a stake in the outcome.
Mixed. Some independent signal (teacher reports, third-party reviews) alongside vendor data.
Vendor-dominated. Mostly published or funded by the company that benefits from adoption.
Vendor only. All evidence is the vendor’s own marketing materials and testimonials.

Ready to apply the Evidence Test with other educators?

The free Harvest Kernel community runs this test on real tools every week. Compare notes with educators, professionals, and lifelong learners who refuse to accept marketing copy as evidence.

Join the Free Community

The Evidence Test

The Evidence Test

Ready to apply the Evidence Test with other educators?

Quick Links

Connect