2
$\begingroup$

My dog show predictive tool is having some trouble with its neural net. Broadly, I start with a couple of factors--age, weight, height, breed (which is a set of dummy variables), a subjective cuteness score--and predict whether the animal will win best in a show (a binary variable). I've got a few hundred shows, so tens of thousands of rows. I'm stuck using the nnet package instead of neuralnet, but I can pretty easily create a decent neural net using

model <- nnet(Win ~ ., x=X_training, y=Y_training, size=30, maxit=500)

which works fine. I've tested it against the holdout test data and the AUC and confusion matrix looks good, and I can use it for predicting future winners. The problem is, how do I determine which variable is most important in predicting the outcome for each new dog show? When I run a prediction for future winners, I use the following code

winpred = predict(model, x, type='raw')

which provides a score between 0 and 1 of the likelihood of winning. What I'd also like is some way of identifying for each row which variable--age, weight, height, breed dummies, and cuteness--contributed the most and least to the score in my predictions. If I were doing this for a single dog, I would use LIME or something else based on this article, but unfortunately, I want to run this on a few hundred odd dogs at a time.

Basically, I want to be able to say to dog owners that their pup is doing well because of x, but what's most holding it back is y (not terribly helpful if it's breed, of course). It seems like there isn't a way to do this with neural networks. Are there other machine learning techniques now that would allow this? If not, any suggestions on other models that would work?

I was thinking of using OLS as a linear probability model or logit/probit, perhaps with LASSO or Ridge. I could use my machine learning model to score dogs, but in the background pull from the regression to identify which variables contribute most strongly, positively, and negatively, to the score. Logit/probit are better from a statistical perspective, but is there any easy way to interpret in R/RStudio, for a few hundred dogs at a time, which attribute contributed most strongly to their score?

$\endgroup$
1
$\begingroup$

Neural Networks are notoriously good at performance and bad at interpretability, i.e. it's very difficult (almost impossible) to explain why a particular prediction was made. It's even more difficult to link the prediction with the features, since the NN does a lot of intermediate calculations where all the features play a role.

The notoriously good models for interpretability are the simple ones: a decision tree is very intuitive to interpret, assuming the tree is not too large. A human can actually follow the tree based on the conditions on the features.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.