Investigation reveals Los Angeles schools suspect AI chatbot misused student data

Los Angeles school district investigators are looking into allegations that their $6 million AI chatbot, “Ed,” may have compromised students’ personal information.

An investigation by the Los Angeles Unified School District’s inspector general’s office included a video interview with Chris Whiteley, a former senior director of software engineering at AllHere. Whiteley claimed that the company’s data security practices violated industry standards and the district’s policies.

Despite Whiteley’s prior warnings about data privacy issues with Ed, the school district and state education officials did not respond. Following The 74’s report on Whiteley’s concerns, investigators met with him on July 2 to discuss the allegations, including the inappropriate handling of students’ personally identifiable information.

Whiteley expressed that the officials from the inspector general’s office showed genuine interest in his revelations concerning Ed, AllHere, and the wider implications for AI in education. He noted their surprise at the mishandling of students’ personal information.

The chatbot processed personal data for all students in a household to respond to various queries, even mundane ones. Whiteley highlighted the concerning practice of accessing data about all of a mother’s children to generate a response for just one child’s query.

The Los Angeles Unified School District declined to comment on the inquiry when directed by the inspector general’s office, focusing on technical aspects of data security. Investigators also probed Whiteley on his experiences with AllHere and its financial situation.

Smith-Griffin, the CEO of AllHere, departed the company following financial challenges and layoffs affecting most employees. The district spokesperson confirmed the financial disruption to AllHere, which had already received $3 million for developing Ed and an integrated portal for students and parents.

Amid the collapse of AllHere, scrutiny intensified after Whiteley raised concerns about offshore processing of students’ information. The whistleblower alleged that the chatbot’s requests were sent to servers in various countries, raising further data privacy red flags.

Whiteley criticized the lack of audits by district leaders when implementing new tools like Ed. He emphasized the importance of understanding how third-party companies handle student information, especially for smaller districts with fewer resources.

The district’s position on Ed has evolved, with conflicting statements about its status and future. Despite the chatbot being unplugged in June, Carvalho continued to promote LAUSD’s AI initiatives and data privacy protocols with third-party vendors.

The district confirmed plans to maintain the online portal, suggesting it could generate revenue and benefit other school districts. The ownership of the chatbot remains with the district, which stands to receive royalty payments if other districts adopt the tool.

Temporary disabling of the chatbot was attributed to AllHere’s uncertainty, pending its restoration with enhanced oversight. Whiteley questioned Ed’s future viability, indicating doubts about the company’s prospects following the controversy and financial struggles.