0
$\begingroup$

I am using TF-IDF for text classification and my solution works well according to the performance metric of my choice (F1 macro). To speed up the training process I have used PCA to reduce the dimensionality of the document vectors.

I am using this for a growing set of datasets and the datasets keep changing.

Is there a way to reuse the TF-IDF vectorizer and the PCA transformation across different datasets? (for time efficiency)

My initial idea is to share the vocabulary and documents of the datasets to create a universal TF-IDF+PCA transformation, but I am worried if 1) it would harm the performance of classification on individual datasets and 2) new datasets might have terms not present in the universal vocabulary.

Are there existing solutions for reusing TF-IDF/PCA across multiple corpuses? and/or an actively changing corpus?

Improve this question
$\endgroup$
3
  • $\begingroup$ Welcome to DataScienceSE. Did you remove the least frequent words before the TFIDF and PCA? This would reduce dimensionality much more efficiently than PCA. $\endgroup$
    – Erwan
    Commented Jun 10, 2022 at 22:39
  • $\begingroup$ I doubt that happens. I could safely cut off the terms down to 10000, but reducing the terms from 10000 to 1000 significantly affected the performance metric. $\endgroup$
    – Ali Asgari
    Commented Jun 22, 2022 at 22:52
  • $\begingroup$ It's important to remove the least frequent words because 1) they do not contribute to determining the target and tend to cause overfitting, and 2) this reduces the number of features a lot thanks to Zipf's law (see explanations in this anwer). $\endgroup$
    – Erwan
    Commented Jun 23, 2022 at 7:14

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.