BUFFALO, N.Y. -- A virtual-reality drama by University at
Buffalo researchers -- aimed at transforming the movie-going
experience -- is driving the development of increasingly
"self-aware" computational agents that are able to improvise
responses to the spontaneous actions of human users.
These improvisational computer agents are expected to influence
the development of electronic devices of tomorrow, making them much
more user-friendly because they will be able to respond to the
idiosyncratic needs of each user.
The researchers' virtual-reality drama, The Trial The Trail, is
a brand new type of dramatic entertainment, where instead of
identifying with the protagonist, the audience becomes the
The multidisciplinary team formed two years ago when Josephine
Anstey, assistant professor in the Department of Media Study in the
UB College of Arts and Sciences, was looking for ways to make VR
dramas more believable. At the same time, Stuart C. Shapiro, Ph.D.,
professor in the Department of Computer Science and Engineering in
the UB School of Engineering and Applied Sciences, was seeking
applications to challenge the computerized cognitive agent called
CASSIE that he and his colleagues had developed.
"We started thinking, 'What happens if you put a powerful
artificial intelligence system -- which is what Stu has developed
-- together with drama and stories?'" recalls Anstey. "The
potential seems endless. You can get to the point where you have
virtual characters that can believably respond to humans in
Instead of sitting in a darkened theater and passively watching
a story unfold, human audience members in this interactive drama
stand up, don gloves and headgear and become immersed virtually in
the world of the characters on the screen.
Instead of using a joystick to compete against virtual
characters as in a video game, the actions and utterances of human
users determine how the virtual characters respond, based on an
ever-growing "library" of actions and verbal communications with
which the VR characters are endowed.
"Because of these attributes, we are creating a more intense
psychological drama for users," said Anstey. "We are creating
characters that are more similar to those you'd find in a
The UB team fully explored these concepts in a paper
presented last year entitled "Psycho-Drama in VR."
By necessity, added Shapiro, those characters are computational
agents that must be capable of behaving in sophisticated and very
human-like ways, attributes that also can help take
"user-friendliness" for computers and other electronic devices to
"This is a step in the design and implementation of computer
agents that are aware of themselves and their actions, as well as
the environment they are in," he explained, "so this work is
relevant to any application in which people interact with a device
Shapiro referred to the team's approach toward its virtual
characters as "cognitively realistic."
"We use a kind of computational 'self-perception' so that just
the way that hearing people can pace their speech more effectively
than deaf people, here's an agent that 'hears' computationally and
can respond to what's happening," he explained. "The agent has some
perception of itself and some level of self-awareness."
While other computer scientists are exploring multiple agent
systems, he continued, this project is more demanding because the
agents in the drama must be able to "perceive" themselves and then
respond to the user.
So, as the human user proceeds through the drama, his or her
actions are being recorded computationally over the Internet,
interpreted psychologically and used to prompt the responses by the
In a sense, the computational agents in the drama must improvise
around the human user, who is acting spontaneously without a
This, Anstey says, is a grand departure from the way that humans
experience other types of computational dramas, such as video
"Very few computer games' characters have psychological lives
that you can respond to," she says.
Because of this, the drama is different every time, a factor
that the researchers say is both a more challenging and exciting
type of entertainment, while also more computationally
In the UB drama, the main VR character is a dramatic
representation of CASSIE, the cognitive agent, based on SNePS,
Semantic Network Processing System, a knowledge-representation
system developed over the past several decades by Shapiro and
William Rapaport, Ph.D., UB associate professor of computer
science, and scores of UB graduate students.
SNePS endows a computational agent with the ability to perform
reasoning tasks, make inferences and do belief revision, where it
can correct itself if it obtains additional information that
indicates that it was misled initially.
"Cassie is one of the few intelligent agents that can process
and communicate in natural language," said Shapiro. "You can talk
to her in English."
He explained that one SNePS runs for each agent in the drama,
and they are learning to communicate and interfere with one
"For computer-based fiction to really emerge, there has to be a
lot of work using intelligent agents," said Anstey. "Anybody
interested in the fiction and drama of the future can't ignore
this. Writers will have to build these skills, will have to, in a
way, think more like computer scientists."
In addition to Anstey and Shapiro, the UB team consists of
David Pape, assistant professor in the Department of Media Study in
the UB College of Arts and Sciences; Anthony Ekeh, a graduate
student in computational linguistics; Michael Kandefer, a doctoral
candidate, and Trupti Devdas Nayak, a graduate student, both in the
Department of Computer Science and Engineering, and Orkan Telhan, a
graduate student in media study.