Artificial intelligence-enabled tools can do all sorts of helpful tasks. But AI is trained on existing (human-generated) data and language and can therefore include biases. And its methods rely on probability and can therefore include mistakes. The Department of Education’s Office of Ed Tech has useful explanations and resources in this blog post. Given these risks and the increasing use of AI in education, the Office for Civil Rights issued guidance on Avoiding the Discriminatory Use of Artificial Intelligence. The guidance includes multiple scenarios that OCR advises would likely cause it to open an investigation related to potential discrimination based on disability, sex, race, national origin, or a combination of bases.  

While the examples are relatively straightforward, the guidance is helpful in pointing out places where schools should be alert for potential problems with AI tools and reminding school leaders not to outsource decisionmaking to those tools. AI-powered tools can be useful in brainstorming, drafting, revising, predicting, and identifying patterns, but the results require human oversight. And when mistakes or problems with the outputs are identified, school administrators should take steps to investigate and respond. 

Below are several examples from the OCR guidance that deal specifically with students with disabilities. Note that in each scenario where OCR states it would likely open an investigation, school officials deferred wholly to AI, even in the face of concerns raised by students, parents, and staff members. This approach can easily land a school in trouble, including in situations like these: 

Example 13 – Test Proctoring Software: Software that uses facial recognition technology and eye movement tracking to monitor students for behavior that may indicate cheating inaccurately flags the eye movements of a student with a vision impairment as suspicious.  

Example 14 – Closed-Circuit Captioning: An AI-aided closed-circuit captioning transcription for class lectures does not accurately capture advanced or course-specific terminology.  

Example 15 – Adaptive Assessments: An AI-driven adaptive assessment that uses a student’s speed as a factor in evaluating student performance, inappropriately penalizes students with an accommodation for extra time 

Example 16 – Writing 504 Plans: A generative AI tool for writing Section 504 Plans for students with disabilities creates plans that are not individualized and do not meet the specific needs of the students. 

Example 17 – Content Moderation: Content moderation software alerts the school if any language that violates the student code of conduct is used on school-issued devices. The software flags students for discipline even if a BIP is in place that addresses the use of inappropriate language in another way. 

Example 18 – Class Management and Bullying: An AI-enabled application that monitors classroom noise is used to allow students to track their collective volume, and the teacher provides rewards for appropriate noise levels. When the class is too loud to receive the reward, students bully classmates for disability-related behaviors.   

Example 19 – Universal Screening: An AI-driven application for universal screening for speech and language disorders falsely flags English language learners as students with speech disorders and misses other students.  

Example 20 – IEP Placement Decisions: An internal software program is trained on past IEPs to recommend appropriate placements for current students. Historically, Black students with disabilities had more hours in special education settings. The software continues to recommend more restrictive placements for Black students, even when not warranted by student needs.   

Example 21 – Student Tracking and Well-Being: AI software that electronically tracks how often students sign out for hall passes is used to estimate students’ mental well-being and thus need for counseling support. The program inaccurately flags students with gastrological disabilities. 

Whether a person, team, or machine makes these decisions, if the result disproportionately impacts with disabilities or fails to provide a free and appropriate public education, OCR is likely to ask questions. As the guidance states, “the nondiscrimination provisions of … federal civil rights laws apply to discrimination resulting from the use of AI.”  

Remember, “AI said so” is not an excuse if OCR (or a court!) identifies discriminatory treatment or a hearing officer finds an IEP is not individualized. AI-powered tools are just that, tools. They are not a replacement for educator judgment and expertise. Please reach out to the authors of this post with questions about AI and special education. And see the related post focused on the examples in the OCR guidance related to sex-based discrimination.