Illustration of the AI Screener interacting with children and a teacher in a classroom setting to automate the detection of speech and language challenges.

Overview of the Institute

Nearly 3.4 million children require speech and language services under the Individuals with Disabilities Education Act (IDEA) and are at risk of falling behind in their academic and social-emotional development without timely intervention by Speech and Language Pathologists (SLPs). Unfortunately, there is a significant shortage of SLPs and the COVID pandemic has further exacerbated this gap, making it almost impossible for SLPs to provide individualized services for children.

The NSF and IES AI Institute for Transforming Education for Children with Speech and Language Processing Challenges (or National AI Institute for Exceptional Education, in short) aims to close this gap by developing advanced AI technologies to scale SLPs’ availability and services such that no child in need of speech and language services is left behind. Towards this end, the Institute proposes to develop two novel AI solutions: (1) the AI Screener to enable universal early screening for all children, and (2) the AI Orchestrator to work with SLPs and teachers to provide individualized interventions for children with their formal Individualized Education Program (IEP).  In developing these solutions, the Institute will advance foundational AI technologies, enhance our understanding of children’s speech and language development, serve as a nexus point for all special education stakeholders, and represent a fundamental paradigm shift in how SLPs serve children in need of ability-based speech and language services.

The Institute’s research will lead to advancements in AI, human-AI interaction, and learning science to improve educational outcomes for children with speech and language related challenges. These advancements and solutions can also benefit children without disabilities extending the reach of ability-based learning interventions.  The Institute will engage participants across the Institute and related stakeholders to advance opportunities to meet their unique educational and workforce development needs.

Illustration of the process for students to receive care starting with ad hoc referral and universal screening by the AI screener, eligibility determination and evaluation by SLPs, and individualized ability based intervention facilitated by the AI orchestrator.

The AI Screener is an edge-based solution that will be initially deployed in early childhood classrooms. It will analyze video and audio streams of children’s classroom interactions, derive conventional speech and language measures used by SLPs, and assess novel and hard to obtain automaticity measures.  The AI Orchestrator is a superset of the AI Screener with its main application in the public-school classrooms. It will help SLPs to administer a wide range of evidence-based interventions and assess their effects on meeting children’s individual IEP learning targets.  At the core of the Orchestrator is a robust multi-agent reinforcement learning framework that can evaluate the potential benefits of different intervention practices and recommend those most appropriate for each child.  Both solutions will push significant advances in self-supervised learning to address sparse and noisy data issues, multimodality perception, learning material rewriting and enrichment, and edge AI for real-time processing. The Institute will develop human-centered AI design methodologies to embody the solutions in a form appropriate for children’s learning. Most importantly, Learning Science will not only inform the initial prototyping and validation, but also continually derive unique insights from the field deployed solutions.

Senior Leadership Team

Portrait of Pamela Hadley.

Pamela Hadley, Co-PI
University of Illinois Urbana-Champaign

Portrait of Julie Kientz.

Julie Kientz, Co-PI
University of Washington

Portrait of David Feil-Seifer.

David Feil-Seifer, Co-PI
University of Nevada, Reno

External Advisory Board

Portrait of Brian Graham.

Brian Graham
Ed.D., Superintendent of Schools, Grand Island Central School District

Portrait of Megan-Brette Hamilton.

Megan-Brette Hamilton
Chief Staff Officer for Multicultural Affairs, The American-
Speech-Language Hearing Association

Portrait of Anil Jain.

Anil K. Jain
University Distinguished Professor, Dept of Computer Science & Engineering, Michigan State University

Portrait of Anjali Joshi.

Anjali Joshi
Advisor, Entrepreneur

Portrait of Jennifer Taps Richard.

Jennifer Taps Richard
Co-Owner and Director, SLPath

Portrait of Tracy Sawicki.

Tracy A. Sawicki
Executive Director, The Peter and Elizabeth C. Tower Foundation

Broader Impact Initiative Leads

Portrait of Alison Eisel Hendricks.

Alison Hendricks
Validation of Impacts
University at Buffalo

Portrait of Ifeoma Nwogu.

Ifeoma Nwogu
Community Outreach
University at Buffalo

Portrait of Letitia Thomas.

Letitia Thomas
Broadening Participation and DEI
University at Buffalo

Portrait of Christine Wang.

Christine Wang
Education and Workforce Development
University at Buffalo

Portrait of Wenyao Xu.

Wenyao Xu
Knowledge Transfer
University at Buffalo

Faculty Participants

Portrait of Maneesh Agrawala.

Maneesh Agrawala
Stanford University

Portrait of Changyou Chen.

Changyou Chen
University at Buffalo

Portrait of Karthik Dantu.

Karthik Dantu
University at Buffalo

Portrait of Mark Frank.

Mark Frank
University at Buffalo

Portrait of Nick Haber.

Nick Haber
Stanford University

Portrait of Mark Hasegawa-Johnson.

Mark Hasegawa-Johnson
University of Illinois Urbana-Champaign

Portrait of Yun Huang.

Yun Huang
University of Illinois Urbana-Champaign

Portrait of Heng Ji.

Heng Ji
University of Illinois Urbana-Champaign

Portrait of Windi Krok.

Windi Krok
University of Illinois Urbana-Champaign

Portrait of Hedda Meadan-Kaplansky.

Hedda Meadan-Kaplansky
University of Illinois Urbana-Champaign

Portrait of Carol Miller.

Carol Miller
Penn State University

Portrait of Abbie Olsewzski.

Abbie Olszewski
University of Nevada, Reno

Portrait of Mari Ostendorf.

Mari Ostendorf
University of Washington

Portrait of Alexander Rush.

Alexander Rush
Cornell University

Portrait of Humphrey Shi.

Humphrey Shi
Georgia Institute of Technology

Portrait of Hariharan Subramonyam.

Hariharan Subramonyam
Stanford University

Portrait of Nigel Ward.

Nigel Ward
University of Texas at El Paso

Portrait of Junsong Yuan.

Junsong Yuan
University at Buffalo

Portrait of Shaofeng Zou.

Shaofeng Zou
University at Buffalo

Partner Institutions

An illustrated map of the USA depicting research institutions participating in the NSF AI Institute. Portraits of the faculty participants are displayed for each institution.