I have a model pipeline for finding similar text documents given an input query text. The model is very simple; I have a corpus of documents on which I train a TfIDF model. When a query is input, we can infer its TfIDF vector. Finally, we compare the query's TfIDF vector to all the vectors of the documents in the corpus using cosine similarity, which finds us the "most similar" texts.

My question is, is there a way to incorporate more micro structure features into the pipeline such that similar document retrieval will perform better, and if so how?

By this I mean, can we use Part of Speech tagging, intent recognition or other methods, as extra features in the pipeline. I am looking for more unsupervised methods here as I have a lot of data (1TB) and it is all unstructured and un-tagged / un-labelled, but supervised suggestions will also be welcome. I appreciate that by incorporating new features, we will most likely have to move to a completely different model paradigm as TfIDF and cosine similarities will not work depending on the structure of the new features.

NOTE: I have already performed a significant amount of text pre-processing on the corpus e.g. stemming, tokenizing, word replacement, stop word removal etc.

  • $\begingroup$ Did you try doc2vec from gensim? $\endgroup$ – amirouche Mar 31 '18 at 15:40
  • 1
    $\begingroup$ You could use tf-idf over the named entities recognized by spacy. $\endgroup$ – amirouche Mar 31 '18 at 16:44
  • $\begingroup$ @amirouche yes I am also testing the doc2vec model, thanks for that. I know that is more sophisticated than tfidf, but I would like to test out tfidf for my purposes $\endgroup$ – PyRsquared Mar 31 '18 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.