Get how similarity between the training data and the income data?

Question

I'am trying to use Clustering and Classification methods as SVM using scikitlearn. I'm also studying some outliers/novelty detections

I want something like a semi-supervised model. I want to predict some class labels, however, I cant training my model with some anomalies/new behaviors that are going to happen eventually.

I'm working with motors vibration, I'll train my model with the vibration when they are "ON" and the vibration when they are "OFF".

With the time the vibration of the "ON" state will start to become slightly different because of some defects, however if I just train with "ON" and "OFF" for sure it'll continue to identify that my motor is "ON" because the vibration with defects are more close to the model "ON".

Therefore I would like to identify how far is my new data from the data that I have already trained my model.

For example, 1 - My predict_prob returns ON because has $98$% of chance to be ON and $2$% to be OFF. So far so good, however my DATA is slightly different from the ON mode in the training data, so I would like to measure this difference, like it has $98$% probability to be ON between ON and OFF and it is $95$% similar to the original model, so with the time this number ll be reduced. For example:

1 day: $98$% to be ON $2$% to be OFF and $95$% similar to the training data

2 day: $98$% to be ON $2$% to be OFF and $93$% similar to the training data

3 day: $99$% to be ON $1$% to be OFF and $90$% similar to the training data

4 day: $95$% to be ON $5$% to be OFF and $85$% similar to the training data

Can someone help me with this part "and $85$% similar to the training data"?

Answer 1

I think I got your difficulty. You are trying to detect defective motors whilst there is more than one state in which they are non-defective. I am assuming that since there are several of those motors even when a motor is OFF the sensors still catch "noise" vibrations. You do have two distinct known patterns: ON and OFF, i.e. OFF is not just the absence of vibration. I see two options for this classification (below) but first let us make a small detour:

and 85% similar to the training data

The vast majority of ML techniques are based on distance measures: the training data is used to define a blobs (e.g. distributions) in (possibly very high dimensional) space, one for each class. The classes of new samples are predicted by comparing their distance to the class blobs - the closest blob is the predicted class.

Now, this means that the classification probability is the similarity to training data. Probabilities in the ranges (0.9, 0.1) or (0.8, 0.2) are very similar to training data, whilst probabilities in the ranges (0.6, 0.4) or (0.5, 0.5) are the most likely distinct from training data. This leads us to one solution you may attempt:

---

Understand probabilities as distances

The probabilities (.predict_proba) of models such as SVMs in sklearn are literally distances, in the case of SVMs the distances within the support vectors. In other words, we can understand these as distances between the middle points of the two blobs: ON and OFF. e.g.

This means that assuming two things:

1. That the classes ON and OFF are well separated (there's reasonable amount of distance between them), and,
2. Defective motors will produce less vibrations than motors ON, but more vibrations than motors OFF (I'm a little worried about this assumption but bear with me).

We can say that:

1. Any probability pair akin of (0.8, 0.2) means that we have an ON or OFF motor. In other words, if we have at least one of the classes with a probability higher than, say, 80%, we are close enough to the blobs of training data.
2. If the probability of ON and OFF is somewhere around 50% then we very likely have a defective motor.

One can choose the probability threshold to his liking (be it 80%, 70% or 90%), as a way to adjust for false positives or false negatives.

Issues with this solution

• The assumption that a defective motor will have vibrations between the ON and OFF classes. i.e. if a defect makes the motor vibrate more than the normal ON it will be undetectable with this solution.
• Assumes linearity and same scale of distance from ON and from OFF for defective motors. i.e. classes are of the same size and equally distributed. Such solution will check distances from each class using the same linear scale (the threshold), it may not detect defects that happen close to the smaller class.

---

Use Novelty Detection

You do have two known classes: ON and OFF. But what you are trying to find is whether a motor is defective or not, where we understand non-defective as the fact that the motor is very similar to most ON motors or very similar to most OFF motors.

Forget about the ON and OFF classes and dump both sets of samples of non-defective motors as one single (non-defective) class and into a novelty detection algorithm. In other words, do not differentiate between ON and OFF motors. Say that both cases (ON and OFF) are OK, just anything that is quite far away from any known cases is not-OK. This will look as a bunch of blobs in space and anything far away from these blobs is a defective motor.

I'll suggest starting with an One Class SVM because it has a parameter called nu= which is quite easy to interpret. It varies between 0 and 1 and the closest to zero it is the more the algorithm is permissive of anomalies (defects) away from the blobs.

Note that we have exactly the same points on this picture as in the previous one. The only difference is that this time all of them are red.

This solution will cope with defects that are anywhere, not only in between the states of ON and OFF.

Issues with this solution

• On Class SVM does not have a .predict_proba. To get an estimate of how far away from a blob an anomaly marked as an anomaly is, one must evaluate the .decision_function of the model and keep track of the distances oneself.
• Even if the classes ON and OFF are well separated one will have false positives and false negatives. A very low nu= parameter still cannot go beyond the blobs. Using an Isolation Forest may allow for better tuning but the decision function would be even more complex to get distances from. Another option is to add marginal versions of ON and OFF to the training data but finding those may not be easy.

---

P.S. If one needs to do both classify ON and OFF AND classify defective and non-defective motors, then one should build two separate models.