Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
After given several talks on NN's, I always have a skeptic that wants a real measure of how well the model is. How do you know the model is truly accurate?
I explain the use of test data etc. to evaluate the total error, however, there is always someone who wants to know about the error associated with each weight?
Can anyone enlighten me as to how I can satisfy these types of questions?
It has become a real issue.
Neural networks are essentially a black box, especially big ones. You could know even how it is designed and how it is training, but you really do not know how it is working in the end. In my work lots of people want to understand the model instead of using "black box" models. This is the reason why companies choose to use linear regressions and polynomial models instead of using stronger machine learning algorithms, like LightGBM and Neural Networks.
I never found a true answer to this question. Some engineers are taught that you cannot use models that you cannot understand. Therefore every model that is a "black box" is not usable for them. This means that most of machine learning models are magic and heresy for them. Though take this with a grain of salt, this is my subjective experience. Sometimes as the time passes these people are more willing to use data science methods because it becomes mainstream. They start to trust the methods, because others use them.
The situation is different on the higher level. For high-tier managers it matters less how to interpret the model, but more what results it could give you. They are more willing to try, especially if there is a hype of something, like "artificial intelligence", "data science".
As a result, I could only give you an advice to find some good support higher in the hierarchy of the company. Someone who believes in data science more and who has more power in the company.
In data science community the performance of the model on the test dataset is one of the most important things people look at. Just look at the competitions on kaggle.com. They are extremely focused on test dataset and the performance of these models is really good.
The only problem with performance on the test dataset is that it depends on the data in the test dataset. If in real life you will have completely different data that will be outside of the bounds of the test dataset, then the test dataset will not be able to give a good approximation of the performance of the model in real life.
I once described them (AI/ML methods in general) to a "senior" mechanical engineer (who really didn't get it, so hated it on principle, and was unfortunately in a position of influence):
It's basically a look up table, interpolating between known data points.
Except, unlike other interpolants like 'linear', 'nearest neighbour' or 'cubic', the underlying functional form is determined to best represent the kind of data you have.
So a "more accurate look up table" approach made him feel more comfortable. Although I'm not sure there were really any winners to that particular argument!!!
I know I'm late to the party, but maybe my perspective will add something to this topic.
I think "skeptics" and "believers" will have to learn to coexist. Both have valid points and both can be validly criticized.
As long as we don't know the exact properties of the data set down to its last details, we will never know whether our models are "accurate". In reality we mostly don't know how things work and we don't know what "true" relationships are in the data, or how they change between test set and training set. That means that evaluation metrics like cross entropy or perplexity or unit error rates can only serve to estimate whether some model fits the data better than some other model, but not for estimating how well either model captures the actual underlying phenomena in the data.
Using test data to validate models boils down to saying "I don't know what relationships are in the data or how well my model captures them, but I will approximate the answer to this question empirically". This is fine for some applications and completely unacceptable for others, so depending on what task you're trying to address you might always get some pushback.
The only reasonable way of providing some security IMO is generating synthetic data with planted relationships and exactly controlled entropies and then either A) proving bounds on how well the model approximates the generating process's entropy (which is in general infeasibly hard to do) or B) demonstrating such a bound empirically (and with statistical significance, which in NN research is often also just approximated through several training runs) while accounting for all the variances you might have artificially skewed via your sampling process.
If someone is thinking of highly security-critical applications, they'll likely require much more rigorous arguments that an NN can be adequate than someone who is not thinking of life-and-death questions. As long as you aren't trying to solve security-critical problems with NNs, don't take criticism too hard -- it is justified in the grand scheme of things, though it's of course not always helpful to you personally.
Something that might help you if someone puts you on the spot is pointing out that there are different ways of using models. If an application is security-critical and errors the model makes cannot be fixed, then every error the model makes is critical and it would be fair of you to acknowledge that. However, in many applications the models are used more as a "guide". As long as the errors a model made can be fixed later (or those errors don't matter much), a model can still be very valuable and save a lot of time even if it gets some things wrong.
This answer will still not satisfy everyone. NN results are typically not very actionable. If you've trained some architecture and it outperforms all other models, you'll still not obviously know what next step to take. Because of that, NNs will always be disappointing to some researchers, i.e. those who want to understand the underlying processes as opposed to building systems that can approximate them reasonably well. With those you'll just have to agree to disagree but again, you shouldn't take their skepticism too hard -- they are probably trying to achieve something different from what you are trying to achieve.
I think this question depends on the type of data you deal with. With images, you can always depict heatmaps of the gradients with Grad-CAM for instance. When deploying, you could bolster the skeptic's trust of the model by providing the gradient heatmap along with the prediction probability.However, this is specific to just one problem domain and I'm not aware of a general answer. In fact, there are several papers discussing the trust of models (this for instance).
Yet, my answer to this question in general would be that neural networks are extremely high dimensional functions and no one ever asks to inspect the exact error associated with all weights in regression models for example. Knowing that each individual part of a neural network works mathematically and that they can be combined should be enough.
