I have a model pipeline for finding similar text documents given an input query text. The model is very simple; I have a corpus of documents on which I train a TfIDF model. When a query is input, we can infer its TfIDF vector. Finally, we compare the query's TfIDF vector to all the vectors of the documents in the corpus using cosine similarity, which finds us the "most similar" texts.
My question is, is there a way to incorporate more micro structure features into the pipeline such that similar document retrieval will perform better, and if so how?
By this I mean, can we use Part of Speech tagging, intent recognition or other methods, as extra features in the pipeline. I am looking for more unsupervised methods here as I have a lot of data (1TB) and it is all unstructured and un-tagged / un-labelled, but supervised suggestions will also be welcome. I appreciate that by incorporating new features, we will most likely have to move to a completely different model paradigm as TfIDF and cosine similarities will not work depending on the structure of the new features.
NOTE: I have already performed a significant amount of text pre-processing on the corpus e.g. stemming, tokenizing, word replacement, stop word removal etc.