1
$\begingroup$

Is it a good idea to run PCA on top of attributes obtained with Tf-Idf?

The tf-idf returns a lot of attributes so in that case I believe it is a good idea to run PCA to reduce the number of dimensions.

On the other hand tf-idf seeks to find something very specific to the example, while with the PCA you try to generalize, would you loose some signal with this sequence of operations.

$\endgroup$

1 Answer 1

0
$\begingroup$

It might depend on your downstream task. For classification I have experienced negligible reduction in the performance metric (F1 macro in my case) while improving the speed of the downstream classifier model.

There is a similar report on using PCA on top of transformer based embeddings.

You may loose the dimensions (terms) that do not add much variance to your data (PCA finds a projection to maximize variance), but are crucial for your downstream task (because they are showing up in the minority classes, for example).

PCA may also help you capture correlations between terms (words that happen in the same document often) without the load of using an n-gram tf-idf.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.