I'm working on a project using TF-IDF vectors and agglomerative clustering -- the idea is that the corpus of documents increases over time, and when a new document is added, the mean cosine similarity with each cluster will be calculated to find the best match.
However, I'm under the impression that it is costly and inefficient to re-calculate the TF-IDF vectors of every document each time a new doc is added to the corpus.
I'm trying to discern if it's possible to generate the TF-IDF vectors on an individual basis for each document (see below), and then calculate the cosine similarity by temporarily reshaping/modifying the two vectors.
doc_1 = ['some', 'text', 'here'] doc_2 = ['other', 'text', 'here'] vectorizer = TfidfVectorizer() vector_1 = vectorizer.fit_transform(doc_1) vector_2 = vectorizer.fit_transform(doc_2) # somehow add the features from vector_1 to vector_2 that do not exist there, and vice versa... # this way vectors can be calculated on the fly without having to regenerate vectors with an ever-expanding corpus
It's my understanding that running the sklearn TF-IDF vectoriser returns not only the vectors but a mapping between the unique words in the corpus and the vectors they correspond to, so I think this approach would be possible, but I'm not sure?
Finally, as I'm new to data science, am I approaching this issue horribly? Is there a much cleaner or efficient way of solving the problem of non-parametrical clustering (i.e. no n_clusters) with an ever-expanding corpus.