CLiC: Communicating in Language-Integrated Context

Background:

The goal in this fundamental research program was to explore basic issues in multi-modal communications between people and machines: understanding how ideas are communicated in context, how they are composed from simpler ideas into complex messages, and especially how references in a shared context are interpreted. For example, saying "Add one more" can have different interpretations depending on what is in focus in the context, what should be added to what and where to add it: 'a scoop of ice cream', 'a block to a stack', 'a task to an agenda', 'a note to a measure'.

The program as a whole has three use case domains:

  • Blocks World -- collaborate on building structures with blocks, either in the real world or in a simulated world.
  • Biocuration -- communication between human biologists and machine curators.
  • Collaborative Composition -- two or more participants work together taking turns to create a story.

SIFT and teammates Tufts Medical School and Brandeis University participated in the first two use cases as part of a consortium that included IHMC, OHSU and Harvard Medical School, building a shared architecture for a dialog system.

SIFT Approach:

We developed the reasoning component for the shared architecture that was based on prior work by James Allen's group at IHMC and the University of Rochester. This component, called the 'Behavioral Agent' or BA, was responsible for interpreting the semantic representation of users text in terms of actionable goals or assertions as understood in the current context. This usually meant rendering the meaning as a directive that one of Harvard's or Tuft's biology agents could execute to answer a question or set up a simulation, as well as coordinating clarification dialogs with the user to settle ambiguities.

We teamed with James Pustejovsky's lab at Brandeis University and Brent Cochran's lab at the Tufts School of Medicine. Later, Martha Palmer's group at the University of Colorado, Boulder was added.

Infographic about SIFT's approach to CLiC

Benefits:

  • Generates language for issuing directives, making suggestions, asking clarification questions, and reporting status.
  • Tracks progress toward shared goals.
  • State-of-the-art situated cognition.
  • Plan libraries for identifying problem-solving opportunities.
  • Hierarchical contexts for resolving linguistic references against prior discourse, world entities, implicit entities, or shared goals.
  • Automated planning for intervening and interacting in a changing world.