I am using natural language processing engines such as IBM's Watson Conversation and Microsoft's LUIS, which take a sentence and classify its intent. For example, "i want to buy food" => "Buy_Food".
I am trying to come up with the best set of utterances to train those engines to understand natural language for a given application. My goal is to improve the accuracy of the NLP engine predicting the correct intent.
Suppose I have a few thousand utterances, each with a known intent. I need to come up with a set of just a few hundred best utterances that will be used to create a model in the NLP engine.
What I tried
At first I tried to take a few hundred sentences randomly, create an NLP engine model, and test its accuracy using a few hundred other sentences from my set. Then I would replace one sentence in the NLP engine model, and re-run the test again. At first, Microsoft's LUIS would guess the intent correctly around 90-95% of the time, and after a few dozen iterations the accuracy goes up to 91-96% - very slightly.
I am also thinking about using part-of-speech tagging to not just replace one sentence randomly, but to strive for syntactical diversity of those sentences.
Reason behind doing this
The reason why I am doing all of this is that it seems that dumping more utterance examples into the NLP engine doesn't seem to make it better; in fact, it seems to make it worse.
This is why I am trying to think of a way to create a limited set of utterances that will be used to train the NLP model and improve the rate of how often the engine guesses the correct intent.