5
$\begingroup$

We have a classification algorithm to categorize Java exceptions in Production. This algorithm is based on hierarchical human defined rules so when a bunch of text forming an exception comes up, it determines what kind of exception is (development, availability, configuration, etc.) and the responsible component (the most inner component responsible of the exception). In Java an exception can have several causing exceptions, and the whole must be analyzed.

For example, given the following example exception:

com.myapp.CustomException: Error printing ...
... (stack)
Caused by: com.foo.webservice.RemoteException: Unable to communicate ...
... (stack)
Caused by: com.acme.PrintException: PrintServer002: Timeout ....
... (stack)

First of all, our algorithm splits the whole stack in three isolated exceptions. Afterwards it starts analyzing these exceptions starting from the most inner one. In this case, it determines that this exception (the second caused by) is of type Availability and that the responsible component is a "print server". This is because there is a rule that matches containing the word Timeout associated to the Availability type. There is also a rule that matches com.acme.PrintException and determines that the responsible component is a print server. As all the information needed is determined using only the most inner exception, the upper exceptions are ignored, but this is not always the case.

As you can see this kind of approximation is very complex (and chaotic) as a human have to create new rules as new exceptions appear. Besides, the new rules have to be compatible with the current ones because a new rule for classifying a new exception must not change the classification of any of the already classified exceptions.

We are thinking about using Machine Learning to automate this process. Obviously, I am not asking for a solution here as I know the complexity but I'd really appreciate some advice to achieve our goal.

$\endgroup$
8
$\begingroup$

First of all, some basics of classification (and in general any supervised ML tasks), just to make sure we have same set of concepts in mind.

Any supervised ML algorithm consists of at least 2 components:

  1. Dataset to train and test on.
  2. Algorithm(s) to handle these data.

Training dataset consists of a set of pairs (x, y), where x is a vector of features and y is predicted variable. Predicted variable is just what you want to know, i.e. in your case it is exception type. Features are more tricky. You cannot just throw raw text into an algorithm, you need to extract meaningful parts of it and organize them as feature vectors first. You've already mentioned a couple of useful features - exception class name (e.g. com.acme.PrintException) and contained words ("Timeout"). All you need is to translate your row exceptions (and human-categorized exception types) into suitable dataset, e.g.:

ex_class                  contains_timeout  ...   | ex_type
-----------------------------------------------------------
[com.acme.PrintException, 1                , ...] | Availability
[java.lang.Exception    , 0                , ...] | Network
 ...

This representation is already much better for ML algorithms. But which one to take?

Taking into account nature of the task and your current approach natural choice is to use decision trees. This class of algorithms will compute optimal decision criteria for all your exception types and print out resulting tree. This is especially useful, because you will have possibility to manually inspect how decision is made and see how much it corresponds to your manually-crafted rules.

There's, however, possibility that some exceptions with exactly the same features will belong to different exception types. In this case probabilistic approach may work well. Despite its name, Naive Bayes classifier works pretty well in most cases. There's one issue with NB and our dataset representation, though: dataset contains categorical variables, and Naive Bayes can work with numerical attributes only*. Standard way to overcome this problem is to use dummy variables. In short, dummy variables are binary variables that simply indicate whether specific category presents or not. For example, single variable ex_class with values {com.acme.PrintException, java.lang.Exception, ...}, etc. may be split into several variables ex_class_printexception, ex_class_exception, etc. with values {0, 1}:

ex_class_printexception  ex_class_exception  contains_timeout | ex_type
-----------------------------------------------------------------------
[1,                    , 0                 , 1              ] | Availability
[0,                    , 1                 , 0              ] | Network

One last algorithm to try is Support Vector Machines (SVM). It neither provides helpful visualisation, nor is probabilistic, but often gives superior results.


* - in fact, neither Bayes theorem, nor Naive Bayes itself state anything about variable type, but most software packages that come to mind rely on numerical features.

$\endgroup$
3
  • $\begingroup$ That's really helpful. In our case we have currently hundreds of patterns so we'd end up having vectors with hundreds of features. Do these algorithms behave well in this situation? Other thing, each time a totally new exception comes up we'd need to classify it manually, possibly add new features and launch the train process to get a new model. Is this an appropriate approach? $\endgroup$
    – IsidroGH
    Aug 9 '14 at 9:35
  • $\begingroup$ NB and SVM should work fine, decision trees over hundreds of variables may be hard to visualize and interpret, but I don't expect degradation in accuracy (though random forests are often suggested in such settings). As for new exceptions, standard way is to keep special <UNKOWN> variable, and really add new variables only when percentage of unknowns exceeds, say, 5%. $\endgroup$
    – ffriend
    Aug 9 '14 at 12:09
  • $\begingroup$ Great. Thanks!! $\endgroup$
    – IsidroGH
    Aug 9 '14 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.