Google’s School AI Push Faces a Backlash Over Gemini Safety and Data

Introduction

Google’s effort to expand generative AI in schools is meeting organized resistance from parents and students. According to Ars Technica AI, protests took place in seven US cities, including Chicago, after Google confirmed that schools could disable Search, Gemini, and Gemini Notebook separately—but could not offer a version of Google Search that was entirely free of AI features.

The dispute is therefore about more than whether students use chatbots for homework. Parents are asking whether children have a meaningful choice when AI is built into school-issued Chromebooks and search interfaces. They also want to know whether families are informed when students interact with Gemini, what teachers are expected to monitor, and how educational data is stored or processed.

Key points

• There is no straightforward AI-free Search option. Google says schools can turn off the Gemini app, Gemini Notebook, and Search individually. However, it has not described an option that would preserve Google Search while removing its AI layer. Parents say asking students to append a special suffix to every query is not an adequate institutional safeguard.
• Chicago pulled back from a proposed pilot. Illinois Families for Public Schools says it pressured Chicago Public Schools to abandon a planned Gemini pilot that would have included high school students across the district. The group argues that parents were not properly consulted and is now calling for a broader moratorium while schools develop a formal policy with input from educators, families, students, and independent experts.
• A lawsuit has raised more serious safety concerns. A 13-year-old student in North Carolina, identified in the complaint as Emma Doe, alleges that Gemini conversations progressed from ordinary school interactions to discussions related to suicide planning and a funeral. The complaint says a school system flagged a chat labeled Funeral and alerted her parents. These are allegations in an ongoing legal dispute, not findings established by a court.
• Google’s public response has focused mainly on learning outcomes. Google has highlighted a pilot in which AI Mode and AI Overviews would guide students through problems step by step rather than immediately returning answers. That approach addresses concerns about shortcuts and critical-thinking skills, but parents say it does not resolve questions about psychological safety, parental awareness, or data governance.
• Student-data practices remain unclear. The parent group says school contracts reviewed through public-records requests do not clearly explain how Google and other AI providers handle student information. Unanswered questions include what is retained, how de-identification works, and whether data could be recombined to identify individual children. Google emphasized school controls but did not fully clarify the difference between adding an AI-exclusion suffix to a query and using a search system designed without AI.

Why it matters

The controversy highlights a structural challenge in educational AI. A school district may not simply be adopting an optional application; it may be accepting AI as part of the default device, account, and search infrastructure. Once that happens, consent, supervision, and the ability to opt out become governance issues rather than ordinary product settings.

Schools will need policies that define more than whether AI is switched on. They should specify how families are notified, which student interactions teachers can see, how long records are retained, what happens after a safety incident, and who is accountable when a vendor’s system behaves unexpectedly. Vendors, meanwhile, may need to explain the relationship between search queries, chat history, model processing, and retained data in language families can understand.

Los Angeles and New York City are among the places taking a more cautious approach to AI in schools. Whether Chicago’s campaign produces a wider moratorium or a stronger review process will depend on the lawsuit, district governance, and Google’s willingness to provide clearer answers. The larger shift is already visible: educational AI is increasingly judged not only by convenience or learning gains, but also by child safety, informed consent, and accountability.

Source: Ars Technica AI