I was reading this blog post titled: The Financial World Wants to Open AI’s Black Boxes, where the author repeatedly refer to ML models as "black boxes".

A similar terminology has been used at several places when referring to ML models. Why is it so?

It is not like the ML engineers don't know what goes on inside a neural net. Every layer is selected by the ML engineer knowing what activation function to use, what that type of layer does, how the error is back propagated, etc.

  • 3
    Something a little subtle: The ML engineer knows all of the structure - how many layers, the activation functions, etc. What they don't know is the weights themselves. But a ML model is so determined by its weights that the evaluation of the model with a specific set of weights can't (currently) be interpreted, explained or understood by humans, even expert humans who fully understand the structure. – isaacg Aug 18 '17 at 7:45
  • Slightly relevant: stats.stackexchange.com/a/297476/100456 – Miguel Aug 18 '17 at 9:20
  • 3
    @isaacg - An ML engineer can easily find out what the weights are. The black box has more to do with not knowing why the weights are what they are and what those weights relate to in the real world. Therefore, it is even more subtle. – josh Aug 21 '17 at 8:35
  • Another related question: datascience.stackexchange.com/q/33524/53479 – mapto Jun 27 at 6:21

10 Answers 10

up vote 48 down vote accepted

The black box thing has nothing to do with the level of expertise of the audience (as long as the audience is human), but with the explainability of the function modelled by the machine learning algorithm.

In logistic regression, there is a very simple relationship between inputs and outputs. You can sometimes understand why a certain sample was incorrectly catalogued (e.g. because the value of certain component of the input vector was too low).

The same applies to decision trees: you can follow the logic applied by the tree and understand why a certain element was assigned to one class or the other.

However, deep neural networks are the paradigmatic example of black box algorithms. No one, not even the most expert person in the world grasp the function that is actually modeled by training a neural network. An insight about this can be provided by adversarial examples: some slight (and unnoticeable by a human) change in a training sample can lead the network to think that it belongs to a totally different label. There are some techniques to create adversarial examples, and some techniques to improve robustness against them. But given that no one actually knows all the relevant properties of the function being modeled by the network, it is always possible to find a novel way to create them.

Humans are also black boxes and we are also sensible to adversarial examples.

  • 1
    The logic of decision trees could be followed in theory, but it is often not practical. I do not see where is the fundamental difference with NNs. – Miguel Aug 17 '17 at 22:13
  • BTW I have used and seen used black box in terms of lack of expertise/interest in learning even the basics of the used tool. – Miguel Aug 17 '17 at 22:15
  • 4
    "But given that no one actually knows the function being modelled by the network". That is wrong / phrased bad. If we didn't know exactly which function was modeled, we could neither train them nor use them for prediction. We know exactly which function is modeled. We do not know (all) relevant properties of it. And the function is complicated. But that's a very different statement. – Martin Thoma Aug 18 '17 at 5:42
  • 1
    @MartinThoma agreed and updated. – ncasas Aug 18 '17 at 7:52
  • 1
    (+1) But a nitpick. Logistic regression does not make class assignments, it only attempts to estimate conditional probabilities. Ditto with a properly used classification tree. Class assignments are imposed by humans that need to make decisions, not by the ML algorithms themselves. – Matthew Drury Aug 20 '17 at 5:43

While I agree on ncasas answer in most points (+1), I beg to differ on some:

  • Decision Trees can be used as black box models, too. In fact, I'd say in most cases they are used as black-box models. If you have 10,000 features and a tree of depth of 50 you cannot reasonably expect a human to understand it.
  • Neural Networks can be understood. There are many analyzation techniques (see chapter 2.5 of my master thesis for some which are aimed at improving the model). Especially occlusion analysis (Figure 2.10), Filter visualization (Figure 2.11). Also the Why Should I Trust You? paper (my notes).

Explaining the prediction of a black-box model by fancy occlusion analysis (from "Why should I trust you?"): enter image description here

I would like to point out The Mythos of Model Interpretability. It formulates some ideas about interpretability in a concise way.

Your question

Why are Machine Learning models called black boxes?

How people use it: Because they do not model the problem in a way which allows humans to directly say what happens for any given input.

Personal thoughts

I don't think this notion of a "black box model" makes much sense. For example, think of weather forecasting. You cannot expect any human to say which weather will be predicted if he is only given the data. Yet most people would not say that physical weather models are black box models. So where is the difference? Is it only the fact that one model was generated using data and the other one was generated using insights into physics?

When people speak of black box models they usually say it as if it is a bad thing. But humans are black box models, too. The critical difference I see here is that the class of errors humans make is easier to predict for humans. Hence it is a training problem (adverserial examples on the NN side) and an education problem (teaching humans how NNs work).

How the term 'black-box model' should be used: An approach which makes more sense to me is to call the problem a "black box problem", similar to what user144410 (+1) writes. Hence any model which only treats the problem as a black box - hence something you can put input in and get output out - is a black box model. Models which have insights (not only assume!) about the problem are not black-box models. The insight part is tricky. Every model makes restrictions on the possible function which it can model (yes, I know about the universal approximation problem. As long as you use a fixed-size NN it doesn't apply). I would say something is an insight into the problem if you know something about the relationship of input and output without poking the problem (without looking at data).

What follows from this:

  • Neural Networks can be non-blackbox (whitebox?)
  • Logistic Regression can be a black-box model.
  • It's more about the problem and your insights about it, less about the model.
  • 1
    Thank you. Your answers are always a pleasure to read :) – Dawny33 Aug 18 '17 at 5:47
  • 1
    You're welcome :-) And thank you for the nice words :-) Please take my answer to your question with a grain of salt. I'm not too sure about it either. I don't think there is a definite answer because people use the word without having a definition for it. So on the one hand the usage between people is likely different and on the other hand even a given single person might not use it always the same way. – Martin Thoma Aug 18 '17 at 5:53

It comes down to model interpretability and explainability. Given the output of a simpler model, it is possible to identify exactly how each input contributes to model output, but that gets more difficult as models get more complex. For example with regression you can point to the coefficients, with a decision tree you can identify the splits. And with this information, you could derive rules to explain model behaviour.

However, as the number of model parameters increases, it becomes increasingly difficult to explain exactly what combinations of input lead to the final model output, or derive rules from the model's behaviour. Say in the financial industry when the COO comes over and asks 'so, why did your high frequency trading algo break the economy', he doesn't want to hear how it was built, just why it sent him bankrupt. It will be possible to state how the model was constructed, but it might not be possible to explain what combinations of factors that the model received as input led to the output, and that’s why people are talking about black boxes.

Black box models refer to any mathematical models whose equations are chosen to be as general and flexible as possible without relying on any physical/scientific laws.

Grey box models are mathematical models where part of the equations (mathematical function) comes from physical known laws but the remaining part is assumed general function to compensate for the unexplained part.

White box models are mathematical models completely built on physical laws and understanding of the system, like for example mechanical motion laws (model of aircraft ..etc)

See: https://en.wikipedia.org/wiki/Mathematical_model#A_priori_information

  • Interesting definition! Lets go through some examples: Logistic regression, SVMs, NNs, decion trees are all black box models. Depending on the context, bayesian models can be in all three categories. Weather models are white-box or gray box models. – Martin Thoma Aug 18 '17 at 5:47
  • I've got to disagree with this answer. You're drawing the distinction between empirical models and models based on physical theory. However, either type of model can be white or black box depending on how it is packaged. – Brian Borchers Aug 20 '17 at 4:58
  • The term black box refers to the underlying 'true' system and is related to the model structure selection problem. – user144410 Aug 21 '17 at 8:27
  • "The modern term "black box" seems to have entered the English language around 1945. In electronic circuit theory the process of network synthesis from transfer functions, which led to electronic circuits being regarded as "black boxes" characterized by their response to signals applied to their ports, can be traced to Wilhelm Cauer who published his ideas in their most developed form in 1941..." Source: en.wikipedia.org/wiki/Black_box#History – user144410 Aug 21 '17 at 8:31

A black box, as you may know, refers to a function where you know the signature of the inputs and outputs, but can't know how it determines the outputs from the inputs.

The use of the term is being buzz-worded incorrectly in this case. It may be beyond the writer/author's willingness or capacity to know and understand ML models, but that does not mean it is beyond the willingness or capacities of others. The engineers that create each ML model know exactly how it works and can pull up the decision tree at will and walk it. Just because someone may be too lazy or it may take a while to do so does not mean the information is not readily available for consumption.

ML models are not black boxes, they are clear boxes that are just really big.

In the blog posting cited in the question, the discussion is about the fact that the experts who develop machine learning models in finance can't explain to their customers (financiers with no training in machine learning) how the model makes the decisions that it does.

This brings out a distinction between models that are black boxes because of information that is truly secret (e.g. the coefficients are encoded in a tamper proof FPGA) and models that are open (in the sense that the coefficients are known) but not comprehensible to a particular audience.

This latter kind of "black box" is problematic because customers want to reassure themselves that the model you've constructed has "face validity." With other types of models such as Logistic Regression, it's relatively easy to look at the coefficients and check that they have the expected plus or minus signs- even a mathematically illiterate MBA can understand that.

I think the black box concept as used in this way originates from black box testing in software and hardware Quality Assurance. It is when you either choose not to / or even can't look into and see the inner working of what you are testing. It could be for a reason that it would be

  1. impractical or impossible to peek into it (it is in a sealed environment and we simply can't look into it) - But it might as well be

  2. because there is a larger chance of writing crappy tests if one can see the inside. Larger risk of (with or without intent) "writing tests designed to pass".

Writing the test to fit the thing that is being tested, lowering the chances of actually finding anything.

It would be perfectly possible for a skilled signal engineer to peek into the inner workings of a neural network and check which features are being selected for in a particular training sequence.

ML engineers don't know what goes on inside a neural net

Sorry to contradict you, but it's true. They know how neural networks learn, but they do not know what any given neural network has learned. The logic learned by neural networks is notoriously inscrutable.

The point of using machine learning is usually to learn the rules that a programmer or domain expert would not think of. This is inherently difficult to figure out.

It's analogous to a conventional computer program written with one letter variable names, no comments, no obvious structure, using obscure mathematics, and all by someone who is now dead. You can step through it in a debugger, but it is still far from clear how it works.

Rarely, someone does take the trouble to figure out what a neural network does. For example, the min-conflicts algorithm was discovered by analyzing a neural network trained on the N-queens problem. But it's a lot of work.

  • The same could be said on some linear methods, e.g. PCA, just the formula in DL is more complicated. – Miguel Aug 17 '17 at 22:09

Machine Learning can be rightly considered Black boxes, solutions for the XOR problem using neural networks can be modelled but as the number of inputs grow, so does the complexity and dimensions. If it is too complex to understand and explain, then it is a black box, whether or not we can calculate the results or not

We can only perceive them upto 3 dimensions but this is sufficient because we can extrapolate this upto higher dimensions using the 3d model as a point of reference. We can imagine local minimums, as well as parts of datasets that are partially learnt.

I have toyed with the idea for a while and so I produced animations of neural networks at work and improved my understanding of neural networks. I have produced animations with 1 and 2 hidden layers (3rd is mostly done) and how they learn data.

The animation is slow and the top right animation showing the upper layers is worth watching, you can speed the animations on Youtube if you like, significant changes can be seen on the top right animation with the Blue and Red Mesh at 3:20 Orange and Red mesh at 6 mins and the Blue, Orange and Red mesh at 8:20. The directions of the weight changes are obviously in the bottom left animation

https://www.youtube.com/watch?v=UhQJbFDtcoc

Black box methods are difficult to explain to the "uninitiated." Anybody in finance or other fields can grasp the basics of regression or even decision trees. Start talking about support vector machine hyperplanes and neural network sigmoid functions and you will lose most audiences

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.