I know I'm late to the party, but maybe my perspective will add something to this topic.
I think "skeptics" and "believers" will have to learn to coexist. Both have valid points and both can be validly criticized.
As long as we don't know the exact properties of the data set down to its last details, we will never know whether our models are "accurate". In reality we mostly don't know how things work and we don't know what "true" relationships are in the data, or how they change between test set and training set. That means that evaluation metrics like cross entropy or perplexity or unit error rates can only serve to estimate whether some model fits the data better than some other model, but not for estimating how well either model captures the actual underlying phenomena in the data.
Using test data to validate models boils down to saying "I don't know what relationships are in the data or how well my model captures them, but I will approximate the answer to this question empirically". This is fine for some applications and completely unacceptable for others, so depending on what task you're trying to address you might always get some pushback.
The only reasonable way of providing some security IMO is generating synthetic data with planted relationships and exactly controlled entropies and then either A) proving bounds on how well the model approximates the generating process's entropy (which is in general infeasibly hard to do) or B) demonstrating such a bound empirically (and with statistical significance, which in NN research is often also just approximated through several training runs) while accounting for all the variances you might have artificially skewed via your sampling process.
If someone is thinking of highly security-critical applications, they'll likely require much more rigorous arguments that an NN can be adequate than someone who is not thinking of life-and-death questions. As long as you aren't trying to solve security-critical problems with NNs, don't take criticism too hard -- it is justified in the grand scheme of things, though it's of course not always helpful to you personally.
Something that might help you if someone puts you on the spot is pointing out that there are different ways of using models. If an application is security-critical and errors the model makes cannot be fixed, then every error the model makes is critical and it would be fair of you to acknowledge that. However, in many applications the models are used more as a "guide". As long as the errors a model made can be fixed later (or those errors don't matter much), a model can still be very valuable and save a lot of time even if it gets some things wrong.
This answer will still not satisfy everyone. NN results are typically not very actionable. If you've trained some architecture and it outperforms all other models, you'll still not obviously know what next step to take. Because of that, NNs will always be disappointing to some researchers, i.e. those who want to understand the underlying processes as opposed to building systems that can approximate them reasonably well. With those you'll just have to agree to disagree but again, you shouldn't take their skepticism too hard -- they are probably trying to achieve something different from what you are trying to achieve.