using LSTM encoder decoder framework to generate sentences

Question

i am trying out multiple approaches to generate domain specific data (sentences).For e.g. if i am building a data set to train a chatbot in the travel domain, i would like to take pairs of sentences like "i would like to book a return trip to Philly commencing on the 24th" and "i am traveling to Phialdelphia for a conference for 3 days beginning the 24th" and use these as the enc and decoder inputs. So once the training is complete, i can feed it the first sentence and generate multiple samples, set a threshold (rather than argmaxing it) and have a fairly large corpus of similar sentences that i could train another classifier with. I personally think its a good analogy to machine translation tasks (where the sampled output is another language but in some sense semantically the same sentence)

While i am trying it out, can anyone tell me if they have tried this approach and seen decent results ? Also any recommendations/ pointers on better approaches? thanks a ton!

Answer 1

I don't know about any research on this topic, but I have thought about similar ideas before. Let's assume the decoder is some RNN that outputs probabilites conditioned on the encoder vector and on what it has generated so far, $P(Word|EncodedVector, DecodedSoFar)$. What you can do is a type of beam search (there is research on this), which is basically greedily take the most likely option so far, but keep all the paths in memory while they still have the potential for a likely enough sentence. The probability of a sequence is just the product of all the predictions that we have sampled. Let's say our model has the following properties (we ignore the encoder):

$P(A) = 0.3$, $P(B) = 0.5$, $P(End) = 0.2$

Now, condtioned on one sample:

$P(A|A) = 0.6$, $P(B|A)=0.1$, $P(End|A)=0.3$

$P(A|B)=0.7$, $P(B|B)=0.2$, $P(End|B)=0.1$

Now let's say when we have two samples the probability of ending is always 1.

This means we get the following probabilities:

$P(End) = 0.2$

$P(A) = 0.3 * 0.3 = 0.09$

$P(B) = 0.5 * 0.1 = 0.05$

$P(AA) = 0.3 * 0.6 * 1. = 0.18$

$P(AB) = 0.3 * 0.1 * 1. = 0.03$

$P(BA) = 0.5 * 0.7 * 1. = 0.35$

$P(BB) = 0.5 * 0.2 * 1. = 0.1$

But we don't want to iterate over all possibilities, this grows way too quickly. What we can do is build a tree which we can bound by our threshold. Let's say we only want samples with at least probability $0.15$. Our root node is empty, with probability 1. Our terminal nodes are the ones that have $End$ as the last part of the sequence. We start growing our tree with the most likely scenario, which is $B$, which means we have $P(empty)=1$, $P(B)=0.5$. For our next option, we can either look at another start than B or by expanding upon B. $P(A)=0.3$ but $P(BA)=0.35$ so we first expand $B$. We now have $P(empty)=1$, $P(B)=0.5$ and $P(BA)=0.35$. The next step is the highest probability, staying at 0.35 by ending $P(AB-end)=0.35$. The next step is expanding our root by going to $P(A)=0.3$. The next step is expanding root with $P(End)=0.2$. Our next step would be $P(AA)=0.18$ but this is below our threshold. We know there are no sequences that are more likely because of the way we have built our tree, which means we have all sequences above the given threshold. You can use something similar to generate the top k most likely samples.

EDIT: Be aware that this does bias towards shorter sentences if your model allows for them, because every probability decreases the total probability. I have not really thought about how much of this is a problem, it also depends a lot on your generating RNN. You could think of some 'penalties' for short ones but you want the metric to be monotonically decreasing because else you cannot bound your tree properly.