26.2 Chatbots

IR-based chatbots
What’s the main idea behind IR-based chatbots?

The main idea is to select the appropriate response from a corpus of natural human text to a user’s turn. The IR-based systems can use any retrieval algorithm to choose an appropriate response from the corpus.

What are the two simplest IR methods to retrieve responses?
  1. Return the response to the most similar turn

  2. Return the most similar turn

The first method takes in the user’s query and find the turn t in a conversational corpus C that’s most similar to the user’s query, and return the human response to t in C. The second method would just return the turn t. Both methods require a similarity function. In practice, the second method seems to have worked better. Both methods can be extended by using more features such as feeding in conversational history, or information about the user or sentiment etc.

What modifications needed to be made to the standard encoder-decoder model for the response generation task?
  1. Changing the objective function or modifying the beam decoder to alleviate the problem of repetitive and dull responses

  2. Incorporate hierarchical structure to enable the models to model longer prior context

  3. Use reinforcement learning to train the models to choose responses that make the overall conversation more natural

Evaluating Chatbots
How can we evaluate chatbots?
  1. Human evaluation

  2. Adversarial evaluation – train a “turing-like” evaluator to distinguish between human-generated and machine-generated responses. A strong response generation system have a higher chance of fooling the evaluator

Slot-filling evaluations and BLEU metrics tend to perform poorly and has less correlation with human judgements.

26.3 GUS: Simple Frame-based Dialogue Systems

What is task-based dialogue?

Dialogue systems that help users solve different tasks.

What is a frame?

A frame is a knowledge structure that allows the system to extracts the different intentions from user sentences. It consists of a collection of slots, each of which can take a set of values. The set of frames is known as domain ontology.

A slot specifies what the system needs to know. An example of a slot is of type city (values of different cities like London) or type airline (values of different airlines).

The goal of task-based dialogue systems is to fill the slots in the frame with the fillers the user intends and perform the relevant action.

Why is GUS architecture a production rule system?

The production rule system allows the system to dynamically switch control based on the user’s input and dialogue history. Different types of inputs lead to different productions to be executed.

Ryan

Ryan

Data Scientist

Leave a Reply