I was reading this blog post titled: The Financial World Wants to Open AI’s Black Boxes, where the author repeatedly refer to ML models as "black boxes".
A similar terminology has been used at several places when referring to ML models. Why is it so?
It is not like the ML engineers don't know what goes on inside a neural net. Every layer is selected by the ML engineer knowing what activation function to use, what that type of layer does, how the error is back propagated, etc.
The black box thing has nothing to do with the level of expertise of the audience (as long as the audience is human), but with the explainability of the function modelled by the machine learning algorithm.
In logistic regression, there is a very simple relationship between inputs and outputs. You can sometimes understand why a certain sample was incorrectly catalogued (e.g. because the value of certain component of the input vector was too low).
The same applies to decision trees: you can follow the logic applied by the tree and understand why a certain element was assigned to one class or the other.
However, deep neural networks are the paradigmatic example of black box algorithms. No one, not even the most expert person in the world grasp the function that is actually modeled by training a neural network. An insight about this can be provided by adversarial examples: some slight (and unnoticeable by a human) change in a training sample can lead the network to think that it belongs to a totally different label. There are some techniques to create adversarial examples, and some techniques to improve robustness against them. But given that no one actually knows all the relevant properties of the function being modeled by the network, it is always possible to find a novel way to create them.
Humans are also black boxes and we are also sensible to adversarial examples.
While I agree on ncasas answer in most points (+1), I beg to differ on some:
Explaining the prediction of a black-box model by fancy occlusion analysis (from "Why should I trust you?"):
I would like to point out The Mythos of Model Interpretability. It formulates some ideas about interpretability in a concise way.
Why are Machine Learning models called black boxes?
How people use it: Because they do not model the problem in a way which allows humans to directly say what happens for any given input.
I don't think this notion of a "black box model" makes much sense. For example, think of weather forecasting. You cannot expect any human to say which weather will be predicted if he is only given the data. Yet most people would not say that physical weather models are black box models. So where is the difference? Is it only the fact that one model was generated using data and the other one was generated using insights into physics?
When people speak of black box models they usually say it as if it is a bad thing. But humans are black box models, too. The critical difference I see here is that the class of errors humans make is easier to predict for humans. Hence it is a training problem (adverserial examples on the NN side) and an education problem (teaching humans how NNs work).
How the term 'black-box model' should be used: An approach which makes more sense to me is to call the problem a "black box problem", similar to what user144410 (+1) writes. Hence any model which only treats the problem as a black box - hence something you can put input in and get output out - is a black box model. Models which have insights (not only assume!) about the problem are not black-box models. The insight part is tricky. Every model makes restrictions on the possible function which it can model (yes, I know about the universal approximation problem. As long as you use a fixed-size NN it doesn't apply). I would say something is an insight into the problem if you know something about the relationship of input and output without poking the problem (without looking at data).
What follows from this:
It comes down to model interpretability and explainability. Given the output of a simpler model, it is possible to identify exactly how each input contributes to model output, but that gets more difficult as models get more complex. For example with regression you can point to the coefficients, with a decision tree you can identify the splits. And with this information, you could derive rules to explain model behaviour.
However, as the number of model parameters increases, it becomes increasingly difficult to explain exactly what combinations of input lead to the final model output, or derive rules from the model's behaviour. Say in the financial industry when the COO comes over and asks 'so, why did your high frequency trading algo break the economy', he doesn't want to hear how it was built, just why it sent him bankrupt. It will be possible to state how the model was constructed, but it might not be possible to explain what combinations of factors that the model received as input led to the output, and that’s why people are talking about black boxes.
Black box models refer to any mathematical models whose equations are chosen to be as general and flexible as possible without relying on any physical/scientific laws.
Grey box models are mathematical models where part of the equations (mathematical function) comes from physical known laws but the remaining part is assumed general function to compensate for the unexplained part.
White box models are mathematical models completely built on physical laws and understanding of the system, like for example mechanical motion laws (model of aircraft ..etc)
See here
ML engineers don't know what goes on inside a neural net
Sorry to contradict you, but it's true. They know how neural networks learn, but they do not know what any given neural network has learned. The logic learned by neural networks is notoriously inscrutable.
The point of using machine learning is usually to learn the rules that a programmer or domain expert would not think of. This is inherently difficult to figure out.
It's analogous to a conventional computer program written with one letter variable names, no comments, no obvious structure, using obscure mathematics, and all by someone who is now dead. You can step through it in a debugger, but it is still far from clear how it works.
Rarely, someone does take the trouble to figure out what a neural network does. For example, the min-conflicts algorithm was discovered by analyzing a neural network trained on the N-queens problem. But it's a lot of work.
A black box, as you may know, refers to a function where you know the signature of the inputs and outputs, but can't know how it determines the outputs from the inputs.
The use of the term is being buzz-worded incorrectly in this case. It may be beyond the writer/author's willingness or capacity to know and understand ML models, but that does not mean it is beyond the willingness or capacities of others. The engineers that create each ML model know exactly how it works and can pull up the decision tree at will and walk it. Just because someone may be too lazy or it may take a while to do so does not mean the information is not readily available for consumption.
ML models are not black boxes, they are clear boxes that are just really big.
In the blog posting cited in the question, the discussion is about the fact that the experts who develop machine learning models in finance can't explain to their customers (financiers with no training in machine learning) how the model makes the decisions that it does.
This brings out a distinction between models that are black boxes because of information that is truly secret (e.g. the coefficients are encoded in a tamper proof FPGA) and models that are open (in the sense that the coefficients are known) but not comprehensible to a particular audience.
This latter kind of "black box" is problematic because customers want to reassure themselves that the model you've constructed has "face validity." With other types of models such as Logistic Regression, it's relatively easy to look at the coefficients and check that they have the expected plus or minus signs- even a mathematically illiterate MBA can understand that.
Machine Learning can be rightly considered Black boxes, solutions for the XOR problem using neural networks can be modelled but as the number of inputs grow, so does the complexity and dimensions. If it is too complex to understand and explain, then it is a black box, whether or not we can calculate the results or not
We can only perceive them upto 3 dimensions but this is sufficient because we can extrapolate this upto higher dimensions using the 3d model as a point of reference. We can imagine local minimums, as well as parts of datasets that are partially learnt.
I have toyed with the idea for a while and so I produced animations of neural networks at work and improved my understanding of neural networks. I have produced animations with 1 and 2 hidden layers (3rd is mostly done) and how they learn data.
The animation is slow and the top right animation showing the upper layers is worth watching, you can speed the animations on Youtube if you like, significant changes can be seen on the top right animation with the Blue and Red Mesh at 3:20 Orange and Red mesh at 6 mins and the Blue, Orange and Red mesh at 8:20. The directions of the weight changes are obviously in the bottom left animation
I think the black box concept as used in this way originates from black box testing in software and hardware Quality Assurance. It is when you either choose not to / or even can't look into and see the inner working of what you are testing. It could be for a reason that it would be
impractical or impossible to peek into it (it is in a sealed environment and we simply can't look into it) - But it might as well be
because there is a larger chance of writing crappy tests if one can see the inside. Larger risk of (with or without intent) "writing tests designed to pass".
Writing the test to fit the thing that is being tested, lowering the chances of actually finding anything.
It would be perfectly possible for a skilled signal engineer to peek into the inner workings of a neural network and check which features are being selected for in a particular training sequence.
