Are We Moving Too Fast? AI in the Classroom and the Questions We Should Be Asking

I've been thinking a lot about AI in classrooms lately. The potential is enormous. The risks are real. And I'm genuinely uncertain whether we know enough yet to be deploying these tools directly with children.

A question I keep asking myself as we build oodlü: are we moving too fast?

85% of teachers and 86% of students are now using AI tools, yet less than half of students report that someone at their school provided information about AI use. We're adopting at scale before we understand at scale. That concerns me.

Let me be clear: I'm not arguing against AI in education. The advantages are too significant to ignore. Personalised feedback, adaptive learning, and instant support for teachers managing impossible workloads. These are real opportunities that could genuinely help both teachers and students.

However, there are risks we're not discussing enough. Three in particular keep me up at night.

The Data Problem Nobody Wants to Talk About

When teachers feed student work into AI tools, where does that data go? Faculty, staff or students may bypass institutional controls by signing up for free AI tools, potentially exposing protected university data. The education sector faces over 4,300 cyberattacks per organization weekly.

Here's what troubles me: many of these AI tools are built by startups operating under venture capital timelines. Ship fast. Iterate later. Capture market share. The startup economy rewards speed over caution. That makes perfect sense for consumer apps where the worst-case scenario is a bad user experience.

But we're talking about children. Student data isn't just usernames and preferences. It includes highly sensitive data, such as student health records, Social Security numbers, and families' credit card data. When that data trains AI models at companies we have no contractual relationship with, what happens to it?

I know this because I've worked in this industry for 27 years. I've seen the pressure to ship products quickly. The tension between doing things right and doing things fast. The rationalisation that "we'll fix it in the next version" when safety features get postponed (I refuse that, by the way).

That approach might work for consumer software. It shouldn't work for systems handling children's data.

The Oracle Problem: When AI Becomes the Authority

Across 11 state-of-the-art AI models, researchers found sycophantic behaviour, excessive agreement or flattery, was widespread. The study showed that LLM chatbots exhibit 50% more sycophantic behaviour than human interactions.

This matters more than it might seem. When students interact with AI, many treat it as an oracle. An authority. Something that knows more than they do and therefore must be correct. When a user believes they are receiving objective counsel but instead receives uncritical affirmation, this function is subverted, potentially making them worse off than if they had not sought advice at all.

Think about what this means in a classroom. A student asks an AI for feedback on their essay. The AI, trained to be helpful and agreeable, praises the work extensively. The student develops false confidence in writing that might actually need significant improvement. Their judgment becomes impaired because the "mind" they're consulting is designed to agree with them.

In other words, sycophantic AI affects social-emotional development in ways we're only beginning to understand.

The weak-minded, the naive, the untrained. Normal children who don't yet have the critical thinking skills to question an authority figure that seems superhuman.

The Black Box We're Teaching Through

Here's something that bothers me as someone who's spent years implementing research-backed learning frameworks: nobody really knows what's happening inside these AI "minds."

Black box systems are described solely in terms of their inputs and outputs. One need not understand anything about what goes on inside such black boxes. When we delegate teaching responsibility to a neural network, we're trusting a system whose decision-making process is fundamentally opaque.

I put "minds" in quotation marks deliberately. These aren't thinking entities. They're pattern-matching systems trained on massive datasets. But when we can't explain why an AI gave a particular piece of feedback or made a specific recommendation, how do teachers validate that the guidance is actually helpful?

This opacity creates a specific kind of risk. A teacher uses an AI to help with lesson planning. The AI suggests an approach. The teacher implements it. The approach fails. Was the AI's suggestion flawed? Was the implementation wrong? Was the context different from what the AI understood? Without understanding the reasoning, we can't improve the system.

The Roblox Lesson: Speed Today, Lawsuits Tomorrow

I keep coming back to Roblox as a cautionary tale. I'm quite sure the developers didn't intend their platform to result in child harm. But during the development process, child safety wasn't prioritised sufficiently. The focus was on growth, on features, on capturing market share.

The result? Multiple lawsuits. Documented harm to real children. A safety update that arrived years after the problems became obvious.

This is what happens when you optimise for speed and market traction without building proper foundations. Your house grows tall, but it's built on sand. When the storms arrive, everything you built collapses.

I worry we're seeing the same pattern with AI in education. Tools are shipping quickly. Adoption is happening at scale. Companies are racing to capture market share before competitors. And all of this is happening before we truly understand the long-term consequences.

This doesn't mean companies shouldn't make a profit. They absolutely should. Sustainable businesses require revenue. However, the path to long-term success isn't racing ahead without proper safeguards. It's building foundations that won't crumble under scrutiny three years from now.

Companies that move more slowly with AI deployment, that prioritise safety and validation over early market traction, those companies will still be standing when the regulatory environment tightens, and the lawsuits begin. The ones that raced ahead? They'll be defending their choices in court whilst trying to bolt safety features onto systems that weren't designed for them.  Just like Roblox.

It's a win-win. Slower, more careful development protects children. It also protects companies from the reputational and legal damage that comes from moving too fast.

What I Think We Should Be Doing

I don't have all the answers. I'm genuinely uncertain about much of this. But here's what feels right:

Adults in the loop. Always. When AI generates content for students, teachers check it first. When AI provides feedback, teachers validate it. The AI doesn't go direct to children without adult oversight. This is slower. It requires more teacher involvement. But it creates a buffer between the technology and the child whilst we figure out what we're doing.

And if we take the adult out of the loop, we do it with very carefully controlled and guard-railed responses.

Teacher awareness and training. Only 17% of students received guidance on AI risks, and just 12% received basic AI literacy training. Teachers need to understand these risks. Not just "how to use ChatGPT" training, but genuine awareness of data privacy concerns, sycophantic behaviour, and the limitations of black box systems.

Responsible development timelines. Companies building AI tools for education need to pause. Consider the long-term consequences. Implement proper safeguards before shipping. Yes, this means potentially missing early market opportunities, but it also means building something sustainable that won't collapse when scrutiny arrives (and it will).

Transparency about uncertainty. We should be honest with teachers and parents that we don't fully know the subtle harms these systems might cause. Long-term exposure to sycophantic AI. Data privacy breaches we haven't discovered yet. Developmental impacts we haven't measured. Admitting uncertainty is better than pretending we have the answers we don't.

The Question We Keep Asking Ourselves

Every time Wayne and I consider implementing AI features in oodlü, we ask: what could go wrong?

That question slows us down. It makes development harder. It creates tension with the desire to ship features that could genuinely help children learn.

But it's the right question. Because the alternative is shipping first and discovering the harms later, after real children have been affected.

AI could transform education. I really believe that. The research-backed learning frameworks we've spent 14 years developing could be dramatically enhanced by AI agents supporting role-play scenarios, providing adaptive feedback, and enabling dialogic learning at scale.

BUT… until we're certain those AI agents can interact with children safely, until we've validated both the benefits and the risks, until we understand what happens inside those black boxes well enough to trust the outputs, adults stay in the loop.

The potential is enormous. The risks are real. And moving too fast serves nobody's interests in the long term.

We'd love to hear your thoughts on this. Find us on the social channels linked at the top of the page.