Explainable AI, how did the computer classified Text?

Question

My question is not about explaining the model or the algorithm

like which neurons were triggered and what are the parameters of perceptrons.

I will explain further

The problem

I have medical reports I want to analyse and extract diagnoses from it.

I built deep neural network model and extracted diagnoses with good accuracy 80%

That is good.

Now When patient look at the results and say: hmmmm, your results says I have Corona, how did the computer know?

at this moment there is no way to answer that.

The predictive model gives us the diagnoses not how it was diagnosed.

Requirements

Is there any mechanism or technique that allows us to find how did the computer made this decision?

If patient was classified with diabities, the model shoud identify words that led to the identification.

Results : Diabities

Explanation : 
               the word "Thirst"
               the word "urinating frequently"
               "weight loss"
               "Fatigue"
               "Insulin"

I have been googling with not much luck and I don't where to look and what is the topic name?

Any idea how to do that or at least guidance of where I can find the answer?

Answer 1

There are libraries like ELI5 or LIME, which can provide explanations for text classification, here is a link to an example: https://eli5.readthedocs.io/en/latest/tutorials/black-box-text-classifiers.html#textexplainer

import eli5
from eli5.lime import TextExplainer

te = TextExplainer(random_state=42)
te.fit(doc, pipe.predict_proba)
te.show_prediction(target_names=twenty_train.target_names)