Quick note: Navigating Education has a new home. Same newsletter, now coming from my own site, so there’s nothing you need to do.
Navigating Education | Issue 4: The research on AI in education is a mess
Helping educators and leaders navigate what’s next in teaching and learning.
The research on AI in education is a mess
Patrick O’Neill at Ivy Tech audited 14 of nearly 40 meta-analyses that all concluded AI helps learning. All 14 failed basic standards. They pooled different populations. They mixed outcomes that are not the same thing: knowledge with motivation, engagement, and perceived learning. And they never defined what “AI use” meant in the studies they pooled. Stephen Sawchuk reported the audit in Education Week on November 2, 2026.
Those meta-analyses have been cited more than 3,000 times on Google Scholar. A Nature paper from May 2025 was retracted a year later. Some of the included studies had “impossible” statistics or references that were likely hallucinated.
The Stanford AI Hub for Learning’s 2026 review sharpens the picture further. Of more than 800 studies, Susanna Loeb and colleagues found 20 with strong cause-and-effect designs. None of those 20 involved K-12 students.
O’Neill put it plainly: “The people who are going to do the best research on AI and education probably haven’t published yet.”
Source: The Research on AI in Education Is a Mess (Sawchuk, Education Week, Nov 2, 2026).
Why this is the Evidence Signal’s job
This is exactly why I am launching Evidence Signal. We do not need another feed of “AI improves learning” headlines. We need a place that disseminates the research, helps people understand it, and sorts quality work from weak work.
Use the audit’s own markers as a checklist when you read the next positive meta-analysis:
- Is the AI use defined (tool, task, and context)?
- Is the population one population, not a mash of ages and settings?
- Are the outcomes actual learning, not only motivation or perceived learning?
- Is the design causal enough to support the claim?
If you have not subscribed yet, start here: Introducing the Evidence Signal.
What you can do Monday
While the research base catches up, students are already using chatbots. Sam Wineburg and Nadav Ziv ran a pilot with 117 high schoolers who judged a false chatbot answer with no supporting links. Roughly 3 in 10 said they trusted it. Nearly half said they were “not sure.” Seventeen said they would follow up. None of them did.
Wineburg and Ziv offer four classroom moves that work in any subject:
- Bots aren’t oracles. Show a citation-less answer. Ask where it comes from and whether students would trust it. Then add “cite your sources” to the prompt and have students judge those sources.
- AI’s initial response is a first draft. Shift from “one and done” to “two and through.” Follow up (for example, a pro/con table with links to credible sources) and assess those sources.
- The same question can yield opposite answers. Chatbots are nondeterministic. Have different students ask the same question, word for word, and compare. Unlike a calculator, the answers will not be identical.
- Uncertainty can’t be the final word. Treat “not sure” as something to resolve with the broader web or another check, not as a stop.
Source: Think Your Students Know More About AI Than You Do? Read This (Wineburg & Ziv, Education Week opinion, Oct 7, 2026).
Podcast: Build Courses in an AI Harness | Navigating Education Ep 80
The evidence on AI tools is thin. That is exactly why course design has to lean on what the learning sciences already show.
In Episode 80, I talk about building courses inside an AI harness: Codex, Claude Code, or an open source harness. If you design in an LMS or build your own, that is where most of your design time should go. You can now go from months of course building down to a day or even hours. That is my experience over the last 12 months. Speed only helps if you know cognitive load, formative assessment, feedback loops, spacing, and interleaving. Without that foundation, a fast course is just a fast course.
You can also run fleets of agents and sub-agents to draft iterations, then keep what actually supports learning. Invest in the harness. Keep the learning sciences in the driver’s seat.
From the blog
I Built a Personalized Study System with Claude. Here Is the Science Behind Why It Worked.
June 6, 2026. A practical walkthrough of building a study system with Claude, grounded in the same learning-science filters Episode 80 argues for.
→ Read the post
The K-Shaped AI Divide: Why Knowledge Decides Who Grows With AI
October 27, 2026. My take on why prior knowledge shapes who benefits from AI tools, and what that means for how we teach with them.
→ Read the post
Build Courses in an AI Harness
November 6, 2026. The companion post to Episode 80: one-module workflow, harness diagrams, and real product screenshots, with learning sciences first.
→ Read the post
Close
11 books, with a twelfth and thirteenth coming in the next year.
Thanks for reading. See you next Tuesday.
– Matt