11
$\begingroup$

I'm looking at pybrain for taking server monitor alarms and determining the root cause of a problem. I'm happy with training it using supervised learning and curating the training data sets. The data is structured something like this:

  • Server Type A #1
    • Alarm type 1
    • Alarm type 2
  • Server Type A #2
    • Alarm type 1
    • Alarm type 2
  • Server Type B #1
    • Alarm type 99
    • Alarm type 2

So there are n servers, with x alarms that can be UP or DOWN. Both n and x are variable.

If Server A1 has alarm 1 & 2 as DOWN, then we can say that service a is down on that server and is the cause of the problem.

If alarm 1 is down on all servers, then we can say that service a is the cause.

There can potentially be multiple options for the cause, so straight classification doesn't seem appropriate.

I would also like to tie later sources of data to the net. Such as just scripts that ping some external service.

All the appropriate alarms may not be triggered at once, due to serial service checks, so it can start with one server down and then another server down 5 minutes later.

I'm trying to do some basic stuff at first:

from pybrain.tools.shortcuts import buildNetwork
from pybrain.datasets import SupervisedDataSet
from pybrain.supervised.trainers import BackpropTrainer


INPUTS = 2
OUTPUTS = 1

# Build network

# 2 inputs, 3 hidden, 1 output neurons
net = buildNetwork(INPUTS, 3, OUTPUTS)


# Build dataset

# Dataset with 2 inputs and 1 output
ds = SupervisedDataSet(INPUTS, OUTPUTS)


# Add one sample, iterable of inputs and iterable of outputs
ds.addSample((0, 0), (0,))



# Train the network with the dataset
trainer = BackpropTrainer(net, ds)

# Train 1000 epochs
for x in xrange(10):
    trainer.train()

# Train infinite epochs until the error rate is low
trainer.trainUntilConvergence()


# Run an input over the network
result = net.activate([2, 1])

But I[m having a hard time mapping variable numbers of alarms to static numbers of inputs. For example, if we add an alarm to a server, or add a server, the whole net needs to be rebuilt. If that is something that needs to be done, I can do it, but want to know if there's a better way.

Another option I'm trying to think of, is have a different net for each type of server, but I don't see how I can draw an environment-wide conclusion, since it will just make evaluations on a single host, instead of all hosts at once.

Which type of algorithm should I use and how do I map the dataset to draw environment-wide conclusions as a whole with variable inputs?

$\endgroup$
  • $\begingroup$ Why aren't other classification methods appropriate here? Sounds like you are concerned with multiple cases leading to a reduced set of outputs. But this is basically every case of representation learning in classification. Neural nets will help you sort out what interaction effects are predictive when base features are not too predictive. Otherwise you could use other methods. $\endgroup$ – cwharland Sep 13 '14 at 19:03
  • $\begingroup$ I welcome any solution. Neural net was just the one I was attempting to utilize. $\endgroup$ – Matt Williamson Sep 14 '14 at 20:08
  • $\begingroup$ Interesting problem. Since you posted this over 6 months ago can I get you to confirm that you're still interested in this before I spend time taking a stab at it? $\endgroup$ – Hack-R Apr 13 '15 at 13:40
  • $\begingroup$ My initial thought would be to use a random forest of logistic regression on per-server-type models. Then you have your benchmarks and you'll find out pretty quick if a neural net will give you more. Neural nets don't always give the best results. $\endgroup$ – user1269942 May 19 '15 at 17:24
2
$\begingroup$

In my opinion you're looking into the wrong methods to solve your problem.

You have strictly no numeric data.

Statistics based machine learning has a very hard time with such problems. Your problem sounds more like one of the problems which should be solved with rule based systems. My first instinct would be to try to understand the rules and put them into code leading to a classification.

There are however methods for learning such rule based systems based on logic. They are only quite unfashionable for machine learning today. https://en.wikipedia.org/wiki/Rule-based_system

$\endgroup$
  • $\begingroup$ +1, If you can easily map out if/thens to tackle a problem there is no way that ANN is going to outperform pure inferential logic. Even a statistical argument (service A is down on 5/6 servers, thus service A is down) is better than dealing with the overhead that a Neural Net requires. $\endgroup$ – Derek Janni Mar 4 '16 at 7:09
2
$\begingroup$

In line with the above comment, I suggest you try a rule-based approach. For each server you have, query its services. If all of the services are down on a server, then you have a server problem. For each service, if no server reports that the service is running, then you have a problem with the service. In the case that it's both, you'll get notices for each and then be able to go in an inspect what's going on with any of the constituents.

The cost, maintenance, and risk of bad results using an ANN model all exceed the simple, rule-based solution and your boss will probably pat you on the back for just doing what makes sense here.

If you're really serious about keeping your servers and processes functional, I suggest you invest in a APM service that gives you reliable, real-time notifications on what's going on in your production environment.

In the case that you're just trying to learn how ANN work - try a different problem. Any well-known dataset for classification or anomaly detection will provide you with a lot more insight into how ANN work than a custom dataset, which can be horribly hard to coerce into an effective learning scheme.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.