When users interact with chatbots, there’s always an intent where the users set out to achieve something, whether that’s to book travel plans or ask for restaurant recommendations. It’s crucial for AI to identify these users’ intent correctly as the solutions that the chatbots will provide will be based on its understanding of the user’s intent.

The problem is that in most cases, you wouldn’t know how many intents exist within your dataset. In today’s post, we looked into how Airbnb aims to solve this problem.

Airbnb’s solution to the above problem has two steps:

  1. Intent detection using LDA

  2. Supervised learning to train a multi-class intent classifier (using CNN)

Intent detection

LDA is a probabilistic model that assumes that each message can be decomposed into a set of topics. LDA assumes that words that are common and similar to each other falls into the same unique topic. With LDA, you would need to predefine the number of topics (a hyperparameter). We can find the optimal number of topics by selecting the number of topics that gave us the highest coherence score.

Future work could explore BERT and doc2vec which takes into account the sequential nature of text.

Multi-class intent classifier using CNN

Training an intent classifier requires us to have dataset with high quality labels and this is expensive to come by. We could use the intents detected in the previous step but this might not include all the intentions in our dataset. The solution proposed by Airbnb is to build the dataset iteratively, starting from intents from the previous step and use product feedback to generate the final set of labels. We would perform manual labelling and inter-rater agreement for each intent label. 13% of messages has multiple intent.

Why CNN over RNN? It’s faster in training and inference time with similar accuracy results. The two steps approach was able to generate better results than step 1 alone.

Two main challenges
  1. Human errors in labelling

  2. Label ambiguity

Ryan

Ryan

Data Scientist

Leave a Reply