4
$\begingroup$

Working on some applied machine learning problems, I've started to encouter some practical difficulties. Those difficulties relate to - but are not limited to - convergence of the learning process, stability trough recalibration, explainability, stability of the explainability trough recalibration. Those problems are a bit difficult to handle, notably because:

  • They are not really dealt with in introductory theoretical books. (However, there are some advanced resources to deal with them individually.)

  • They are mostly specific to advanced ML models, compared to more standard approaches.

  • You only get aware of them once you stumble upon them by doing some very specific tests.

I would be interested in some serious sources on the main practical problems related to machine learning (not necessarily how to deal with them). Both as a checklist of pitfalls to avoid and as an authoritative argument if I have to formulate some concerns about any future ML solution. Can you help me with any sources?

$\endgroup$
2
  • $\begingroup$ This seems awfully broad for an SE question... You probably should narrow it down or split it up into several narrower questions. $\endgroup$
    – Avatrin
    Mar 5 '20 at 19:58
  • $\begingroup$ I don’t want to deal with each one in a single question. I’d like to find a serious source on what to watch out for. $\endgroup$
    – lcrmorin
    Mar 5 '20 at 20:04
1
$\begingroup$

How does one even start describing these practical limitations?

Most authoritative source on the main practical (and theoretical) problems related to machine learning is formal framework describing it. Mathematics.

Right here we have a problem, even tough you look at all of these ML algorithms and you conclude its math, we do not explain the whole process of a ML problem. I cant find a citation but Bengio or someone said that computer science is hard science because you have mathematical underpinning for everything, and ML is more of a soft science where you learn by trying, but only after you tried. (Not always the case ofcourse, thats why everyone is researching ML now to give it some structure)

Take a simple neural network. You know its matrix multiplication, backpropagation bla bla. Great. But what are the topological properties of certain architecture, convergence criteria, what functions can it approximate. Some of these questions are known and/or are being researched. Lets look at this one for example: What functions can we guarantee (in a formal sense) that a NN can approximate and under what conditions?

Well we can express a task as an optimization one. And in order to converge to optimal solution, under certain constraints, we need to satisfy certain assumptions.

Regarding DNN (deep neural networks) and mathematical theory behind it , convergence assurance is given with famous Universal Approximation theorem that states that every smooth function can be estimated given enough parameters.

Caveat just because we can do it in theory does not mean its possible. For example approximating a function that generates random numbers would require infinite recources




But what about non-smooth functions (such as Time-series) one?

Well the TL;DR of the DNNS FOR NON-SMOOTH FUNCTIONS is that for a special set of piecewise smooth functions "convergence rates of the generalization by DNNs are almost optimal to estimate the non-smooth functions"

What is piecewise smooth function? function whose domain can be partitioned locally into finitely many "pieces" relative on which smoothness holds, and continuity holds across the joins of the pieces.

enter image description here

Ok but WHY can a DNN approximate these types of functions?

" The most notable fact is that DNNs can approximate non-smooth functions with a small number of parameters, due to activation functions and multi-layer structures. A combination of two ReLU functions can approximate step functions, and a composition of the step functions in a combination of other parts of the network can easily express smooth functions restricted to pieces. In contrast, even though the other methods have the universal approximation property, they require a larger number of parameters to approximate non-smooth structures"

Conclusion there is a mathematical theory that insures approximations of a set of certain non-smooth functions using DNN. So if we have non-smooth function that satisfies these constraints, we can find an optimal architecture and get optimal convergance rates.

Concluding your question There are best practices that constantly evolve and you can get a check-list that just isnt relevant (Take computer vision problems, check list 2 years ago isnt same as today). BUT what is constant and remains the best authority is formal under pinnings such as mathematics. It can tell you directly the "best practice" of when it would be futile to even try to approximate a function.

$\endgroup$
0
$\begingroup$

I think I found what I had in mind at the time. The book Machine Learning Engineering by Andriy Burkov (very good book overall) gives a list of problems that cannot be solved with ML or in other words, some practical limitations of ML. The book being accessible on a “read first, buy later” principle (see here) I copy paste the interesting part:

Chapter 1.5 When not to use Machine Learning.

There are plenty of problems that cannot be solved using machine learning; it’s hard tocharacterize all of them. Here we only consider several hints.You probably should not use machine learning when:

  • every action of the system or a decision made by it must be explainable,
  • every change in the system’s behavior compared to its past behavior in a similarsituation must be explainable,
  • the cost of an error made by the system is too high,
  • you want to get to the market as fast as possible,
  • getting the right data is too hard or impossible,
  • you can solve the problem using traditional software development at a lower cost,
  • a simple heuristic would work reasonably well,
  • the phenomenon has too many outcomes while you cannot get a sufficient amount ofexamples to represent them (like in video games or word processing software),
  • you build a system that will not have to be improved frequently over time,
  • you can manually fill an exhaustive lookup table by providing the expected outputfor any input (that is, the number of possible input values is not too large, or gettingoutputs is fast and cheap).
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.