Beyond Logins: What the Department of Education’s New Edtech Guidance Gets Right and What It Misses

The U.S. Department of Education just released its long-awaited guidance on education technology in schools. The Dear Colleague Letter, authored by Assistant Secretary Kirsten Baesler, tells states and districts to stop evaluating edtech by screen time and engagement metrics and start asking whether tools actually improve learning. This is the right direction, but the guidance stops short of telling districts how to measure learning over time.

“Responsible design is the floor,” Baesler writes. “Not only do we need to know if the tool works, but if it works, why does it work, for whom does it work, and under what conditions” (U.S. Department of Education, 2026). The guidance encourages districts to move beyond adoption metrics toward performance-based contracting and outcomes-focused procurement. Five states, including Arkansas, Indiana, Louisiana, Michigan, and Texas, are already exploring these models.

Adoption metrics tell you whether a tool was purchased and whether students logged in. They do not tell you whether learning happened. The Department of Education is now making the same argument at the federal level. But the guidance leaves the measurement question largely unanswered. It says to evaluate outcomes. It does not say how.

The Major Gap: Measuring Learning Over Time

Here is where I want to push beyond the guidance. Knowing whether a tool works is not a one-time question. It is a longitudinal one. A student’s interaction with an adaptive math platform in September should tell a different story from that in February, and both should tell a different story from that in May. If districts are serious about measuring learning outcomes, they need tools and practices that capture growth over time, not just snapshots.

This is where the science of learning becomes essential. Spaced practice is one of the most reliably replicated phenomena in cognitive science, with a quantitative synthesis of 317 experiments showing that distributing practice across sessions strengthens long-term retention (Cepeda, Pashler, Vul, Wixted, & Rohrer, 2006). Interleaving, mixing problem types within sessions, has a smaller and more conditional evidence base. The strongest meta-analytic signal comes from category learning with visually similar materials, and for mathematical tasks, the effect is small but positive (Brunmair & Richter, 2019). The technique does not transfer cleanly to all subject domains. Both strategies are temporal, but the measurement window depends on what is being assessed. Spacing effects can appear within days for short intervals and across weeks or months for longer ones (Cepeda et al., 2008). If a tool claims to support durable learning, it needs to produce data that shows whether that learning is actually durable over the interval that matters for that skill.

Yet most edtech dashboards show usage frequency, completion rates, and time on task. These are input metrics. They tell you what the student did, not what the student learned. A student using an adaptive platform for 30 minutes, three times a week, for eight weeks shows impressive usage data. But if the platform is not spacing content in ways that foster durable learning, its use is instructional noise.

What Better Measurement Looks Like

Think about what a digital mini whiteboard does in a classroom. A teacher poses a question, every student responds, and the teacher sees the full distribution of answers in seconds. That is a formative assessment snapshot of student performance in real-time. Now imagine doing that repeatedly across a unit and across a semester. Each snapshot becomes a data point in a longitudinal story about what students know and where they are still struggling. A single snapshot is low-reliability on its own. The value comes from the pattern across multiple snapshots over time.

This is what adaptive edtech could be doing, but often does not. The best adaptive software does not just adjust difficulty in the moment. It spaces content over time, tracks retention across intervals, and shows teachers whether students are maintaining what they learned weeks ago or forgetting it. That is the data the Department of Education is asking districts to demand from vendors, even if the guidance does not name these specific mechanisms.

The distinction between quantitative and qualitative measurement matters here. Quantitative data, response accuracy, retention curves, and error patterns can show whether students are improving on specific skills over time. Qualitative data, student explanations, and work samples tell you whether that improvement translates into deeper understanding. You need both. A tool that only produces quantitative metrics gives you a partial picture. A tool that only produces qualitative artifacts gives you no scalable way to track patterns across a class or a district.

What Districts Should Actually Demand

The Department of Education’s guidance encourages districts to expect vendors to publish independent evaluations, share implementation guidance, and provide transparent information about product limitations. These are good expectations. But districts should go further.

Ask whether the tool spaces and interleaves content. If a platform presents content in blocked sequences where students practice one skill at a time and then move on, it is not designed around the science of learning. Spaced practice should be built into the adaptive algorithm.

Ask what longitudinal data the tool produces. Can a teacher pull a retention curve over four weeks, eight weeks, or a semester? Can the tool show whether students who learned a skill in October can still demonstrate it in March? If not, the tool is generating usage data rather than learning data.

Ask whether the tool captures both quantitative and qualitative evidence. Does the platform produce only numeric scores, or does it also capture student reasoning and work products? The most useful tools layer both.

Ask how the tool supports formative assessment cycles. Digital mini whiteboards, quick-response tools, and adaptive platforms that embed low-stakes retrieval practice give teachers repeated snapshots. Over time, those snapshots become a trajectory. Tools that only assess at the end of a unit give you a single data point.

Ask whether the vendor has outcome data, not just engagement data. Districts should ask vendors directly: Do you have evidence that students who used your tool learned more than students who did not? Login frequency is not learning.

The Implementation Fidelity Problem

There is a catch that the guidance does not address. A tool can be evidence-based and aligned with the science of learning and still produce zero impact if implemented at 40% fidelity across a school or district. In districts where adaptive platforms have been piloted across multiple schools, vendor efficacy reports sometimes show strong effect sizes while the district’s own analysis finds near-zero impact. The difference is not the tool. Some schools used the spaced-review module faithfully, while others used the platform as quiet time. Districts that want to measure learning outcomes from edtech must pair vendor demands with internal fidelity measurement, including observation protocols, usage-pattern audits, and coaching. Without fidelity data, outcome data is uninterpretable. You cannot tell whether the tool failed or the implementation failed.

The Real Test vs. Guidance

The Department of Education’s guidance is a step in the right direction. It signals a move from screen time to instructional value, from adoption to outcomes, and from vendor claims to evidence. But the guidance is deliberately hands-off on implementation. It tells states and districts to figure out the details themselves.

That is both the strength and the limitation of this guidance. It sets the floor. It does not build the ceiling. Districts that want to do this well need concrete frameworks for measuring learning over time, rubrics for evaluating whether edtech tools produce longitudinal evidence, and professional learning that helps educators interpret that evidence.

Ultimately, the question is not whether schools use edtech. The question is whether the edtech schools’ use helps them see learning clearly over time, not just in the moment. If a tool cannot show teachers what students retained last month and what they will still know next semester, it is not measuring learning. It is measuring activity. And the Department of Education, to its credit, is now saying the same thing.

The districts that take this seriously will be the ones that do not just demand better tools. They will demand better evidence. And they will have the frameworks to know the difference.

What Teachers Can Do Tomorrow

Most of this piece has been aimed at people who evaluate and procure edtech. But the people who use these tools with 28 students during fourth period also need something concrete. Here are three moves any teacher can make.

First, look at the dashboard differently. If your platform shows only logins, time on task, and completion rates, then that is an input dashboard. Ask your tech director whether the platform has a retention or growth report you have not seen.

Second, track one skill over time yourself. Pick one skill and use whatever formative tool you have, a digital whiteboard, an exit ticket, or a quick-response app, to check it weekly for four weeks. You do not need a platform to build a longitudinal picture. You need a consistent question repeated across time.

Third, flag fidelity issues upward. If you are expected to use a tool three times a week but the schedule only allows once, say so. Implementation fidelity starts with the realities of a teacher’s weekly schedule.

What is one edtech tool you currently use that you could evaluate differently starting tomorrow?

References

• Brunmair, M., & Richter, T. (2019). Similarity matters: A meta-analysis of interleaved learning and its moderators. Psychological Bulletin, 145(11), 1029-1052. https://doi.org/10.1037/bul0000209
• Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin, 132(3), 354-380. https://doi.org/10.1037/0033-2909.132.3.354
• Cepeda, N. J., Vul, E., Rohrer, D., Wixted, J. T., & Pashler, H. (2008). Spacing effects in learning: A temporal ridgeline of optimal retention. Psychological Science, 19(11), 1095-1102. https://doi.org/10.1111/j.1467-9280.2008.02209.x
• U.S. Department of Education, Office of Elementary and Secondary Education. (2026, August 20). Guidance on education technology and screen time in schools. Dear Colleague Letter. https://www.ed.gov/media/document/guidance-education-technology-and-screentime-schools-114386.pdf