Protecting Privacy as You Chat

The use of virtual assistants and chatbots is on the rise as more people make use of artificial intelligence at work, school and home. But as adoption of these tools increases, privacy risks mount as well.

According to new research from the University at Buffalo School of Management, seemingly harmless prompts and casual questions can share more private information—such as intentions, travel, opinions and personal circumstances—than users realize. 

“Proactively alerting users to potential privacy exposure during interactions with large language models has become an urgent and practical need,” says study co-author Shaojie Tang, professor and chair of the Department of Management Science and Systems. “Because these tools are tied to user accounts, every prompt is connected to a real person.”

The researchers’ paper, accepted by the leading data science and AI conference KDD 2026, examines how privacy-detection models can be developed for real-world AI interactions.

To explore whether users could be warned before unintentionally sharing private details about themselves, the researchers constructed the first large-scale multilingual dataset of real AI conversations using nearly 250,000 user queries and more than 150,000 annotated privacy phrases. They then used an advanced AI system to analyze the data step-by-step.

First, the system determined whether a message revealed anything private about the user, such as plans, preferences or work details. Next, it identified the exact words in the message that caused the privacy risk and generated a brief explanation of what those words revealed. The final dataset was used to train and test smaller, privacy-friendly AI tools that could one day warn users before they hit send that their prompt might share more than they intend.

The study is preliminary but provides a foundation for the development of AI systems that can provide timely alerts on the devices people use daily. 

“With the right training, smaller AI models running on personal devices can detect privacy risks more effectively than much larger pre-trained cloud-based systems, offering users more control over how much they are sharing,” says Tang.