Feature importance after PCA (or other dimensionality reduction methods)

Question

I have text data which I one hot encoded and then used PCA on it (although I'm experimenting with other methods as well, LDA, NMF..). I am using the result of the dimensionality reduction as an input for a supervised classification task.

Now I can use the random forest feature importance or other methods to get feature importance of the input to the supervised cls task. However, naturally these features are meaningless. I would like to know which words are most the most important for this classification. In other words, somehow propagate the feature importance score back through the PCA.

Is there any known method to do it?

Answer 1

Most of the PCA methods return the linear transformation matrix which allows to convert from Component to Variable and viceversa.

What is impossible to do is to assign a feature importance from component to variable (unless you are willing to accept the assumption that "feature importance" could be assigned linearly).

If you accept the idea between parenthesis, you could have a linear equation system $Ax = b$ where $A$ is the component-to-variable matrix (of size components-by-variables), $b$ is the feature importance and $x$ is the result (the variable importance).

This idea I am giving to you is what I think could work, you are likely not going to find it in any research or similar.