Creating the optimal set of utterances to train a natural language processing engine

Question

Context

I am using natural language processing engines such as IBM's Watson Conversation and Microsoft's LUIS, which take a sentence and classify its intent. For example, "i want to buy food" => "Buy_Food".

Problem

I am trying to come up with the best set of utterances to train those engines to understand natural language for a given application. My goal is to improve the accuracy of the NLP engine predicting the correct intent.

Suppose I have a few thousand utterances, each with a known intent. I need to come up with a set of just a few hundred best utterances that will be used to create a model in the NLP engine.

What I tried

At first I tried to take a few hundred sentences randomly, create an NLP engine model, and test its accuracy using a few hundred other sentences from my set. Then I would replace one sentence in the NLP engine model, and re-run the test again. At first, Microsoft's LUIS would guess the intent correctly around 90-95% of the time, and after a few dozen iterations the accuracy goes up to 91-96% - very slightly.

I am also thinking about using part-of-speech tagging to not just replace one sentence randomly, but to strive for syntactical diversity of those sentences.

Reason behind doing this

The reason why I am doing all of this is that it seems that dumping more utterance examples into the NLP engine doesn't seem to make it better; in fact, it seems to make it worse.

This is why I am trying to think of a way to create a limited set of utterances that will be used to train the NLP model and improve the rate of how often the engine guesses the correct intent.

Answer 1

Great question, I've been checking back to see if anyone responds but nothing yet, so I'll do my best.

It seems like me, you have also found out that simply adding more and more training data does not increase the model’s accuracy.

You need to make sure your model contains balanced, representative samples for each intent. Simply picking utterances at random means they may not be represent all of the types of language a user may use.

For example, let’s say you had two intents - “buy_food” for buying some food, and “list_menu” for finding out what is available for order. If you were to heavily populate buy_food with lots of utterances that begin with “I want to”, the NLP engine will associate the phrase “I want to” with buying food, even though someone may quite rightly say “I want to see what’s on the menu”. To overcome this, make sure you balance out each intent’s utterances and don’t focus on phrases that aren’t specific to the intent. In this case, you could counterbalance by having some utterances in your “list_menu” intent such as “I want to see what’s on the menu”.

As a rule, for each intent I would try and include up to 5 or 6 ways to say something, and then for each of these, include 4 or 5 variations of this, leaving you with about 25 samples for an intent in total.

There is also the question around how you plan your intents. For example, do you have a “buy_food” intent and a “buy_drinks” intent, or should you simply have a “buy” intent, and then have the thing they want to buy as an entity? (think of your intent as a method, and your entity as a parameter) My suggestion would be to keep your intents broader and have the specifics handled by entities. This will help avoid confusion between intents.

I recommend checking out a tool called QBOX. It connects to LUIS/Watson/DialogFlow and lets you score and visualise intents so you can see any possible confusion between them. This should help you to plan your intent structure and visualise the confusion between your intents, and know if you're causing regressions as you expand your model. (Disclaimer: this is a product I am working on, and built precisely because of the type of problem you're having, so I've not linked to it, but if you Google the name along with the term 'chatbot', you should find it!)