Data set contains records of short text, typically a sentence. The goal is to find duplicated records and similar records. Currently, I have tried R package 'text2vec', the glove word vectors and the similarity APIs provided by the package.
There is a smaller subset of this data which is already tagged as duplicated. Currently, I have not factored in this as part of model training. Also, using the text2vec package, the results are not great on this test set. So now I am considering RNNs which are known to perform well in text similarity.
Now, I need help in feature engineering and preparing my input layer. Sentences S1 and S2 that need to be compared differ in length (the word representations differ in their dimensions). How to normalize this difference? Should I consider bag-of-words or glove word vectors (word vectors being much superior representations)? Any inputs in this regard will help.