I have trained fastText model and some fully connected network build on its embeddings. I figured out how to use Lime on it: complete example can be found in Natural Language Processing Is Fun Part 3: Explaining Model Predictions
The idea is clear - put 1 sentence into Lime, it drop words and generate some new sentences from my and check how score changes.
My next idea - use SHAP values for this.
SHAP values can be used for any deep model, using
DeepExplainer. Here is a usage example: Keras LSTM for IMDB Sentiment Classification
But I can't use it for my ensemble, because
DeepExplainer needs tensors as input, but I want to fed sentences. I don't want to use BoW or TF-IDF for that - I loose fastText power in that situation.
What I want to achieve - get some shap-plots built on words of my sentence.
Is it possible?