Need term or method name for evaluation of CNN without ground truth using e.g. a regression model

Question

I have the following problem, I have trained a CNN and I can evaluate the network in-sample. I want to use the trained model for the class prediction of images for which I have no ground truth. However, there are other features referenced to these images that I can implement in a regression model along with predicted labels to predict Y. The only way to evaluate somehow the CNN is to infer if the predicted labels have an effect on Y e.g. to evaluate the significance of the variable with the predicted classes or the performance of the entire regression model.

Since the regression is interpretable, I think the approach can be assigned to interpretable AI (@Nikos M. thank you). Unfortunately, I cannot assign the approach to any method/taxonomy of ineterpretable AI I have read so far. The regression model serves in my example as a kind of surrogate model. It is not a global surrogate because I do not use the predictions from the black box but use the trained CNN model to predict new unknown images in the "surrogate" model.

Can anyone tell me if this kind of evaluation method for machine learning models has already been described in the community and what is it called?

Tnx

Answer 1

What you ask for is related to what is termed as explainable AI (especially for deep models, like CNNs). Explainable AI methods try to provide (quantitative) insight as to why the model makes this or that prediction. So you can search into this type of literature for approaches.

Effectively what you do with the regression model is trying to quantify how this or that feature affects prediction, which is an approach to explainable AI