# Get how similarity between the training data and the income data?

I'am trying to use Clustering and Classification methods as SVM using scikitlearn. I'm also studying some outliers/novelty detections

I want something like a semi-supervised model. I want to predict some class labels, however, I cant training my model with some anomalies/new behaviors that are going to happen eventually.

I'm working with motors vibration, I'll train my model with the vibration when they are "ON" and the vibration when they are "OFF".

With the time the vibration of the "ON" state will start to become slightly different because of some defects, however if I just train with "ON" and "OFF" for sure it'll continue to identify that my motor is "ON" because the vibration with defects are more close to the model "ON".

Therefore I would like to identify how far is my new data from the data that I have already trained my model.

For example, 1 - My predict_prob returns ON because has $$98$$% of chance to be ON and $$2$$% to be OFF. So far so good, however my DATA is slightly different from the ON mode in the training data, so I would like to measure this difference, like it has $$98$$% probability to be ON between ON and OFF and it is $$95$$% similar to the original model, so with the time this number ll be reduced. For example:

1 day: $$98$$% to be ON $$2$$% to be OFF and $$95$$% similar to the training data

2 day: $$98$$% to be ON $$2$$% to be OFF and $$93$$% similar to the training data

3 day: $$99$$% to be ON $$1$$% to be OFF and $$90$$% similar to the training data

4 day: $$95$$% to be ON $$5$$% to be OFF and $$85$$% similar to the training data

Can someone help me with this part "and $$85$$% similar to the training data"?

• Cross posting is normally frowned upon on SE, that said this is probably the correct site for the question at hand so it should be fine (and you got sent here from there anyway). Now, the question is still quite confusing but we should be able to figure it out. Let me ask you this: is this an exercise in some form of grad course (in which case the objective is to give you understanding) or a real situation - perhaps for work in a non-computational area (in which case i'll ask you to describe more of the real situation, e.g. where the model will be used). – grochmal Jun 12 '19 at 22:48
• This is a real situation. We developed some IoT LoRa sensors to get the Vibrations Data. We have this data in a local DB and we export it to a DB in a EC2 in our cloud. We have any motors with different behaviors in our plant. I would like to record the "good" behavior, after we would like to check if this behavior continues "good" or if it changes, and we would like to keep track this behavior to find the exactly day to do the maintenance instant of do a general maintenance once a month. @grochmal – André Braga Jun 14 '19 at 17:46

I think I got your difficulty. You are trying to detect defective motors whilst there is more than one state in which they are non-defective. I am assuming that since there are several of those motors even when a motor is OFF the sensors still catch "noise" vibrations. You do have two distinct known patterns: ON and OFF, i.e. OFF is not just the absence of vibration. I see two options for this classification (below) but first let us make a small detour:

and 85% similar to the training data

The vast majority of ML techniques are based on distance measures: the training data is used to define a blobs (e.g. distributions) in (possibly very high dimensional) space, one for each class. The classes of new samples are predicted by comparing their distance to the class blobs - the closest blob is the predicted class.

Now, this means that the classification probability is the similarity to training data. Probabilities in the ranges (0.9, 0.1) or (0.8, 0.2) are very similar to training data, whilst probabilities in the ranges (0.6, 0.4) or (0.5, 0.5) are the most likely distinct from training data. This leads us to one solution you may attempt:

# Understand probabilities as distances

The probabilities (.predict_proba) of models such as SVMs in sklearn are literally distances, in the case of SVMs the distances within the support vectors. In other words, we can understand these as distances between the middle points of the two blobs: ON and OFF. e.g.

This means that assuming two things:

1. That the classes ON and OFF are well separated (there's reasonable amount of distance between them), and,
2. Defective motors will produce less vibrations than motors ON, but more vibrations than motors OFF (I'm a little worried about this assumption but bear with me).

We can say that:

1. Any probability pair akin of (0.8, 0.2) means that we have an ON or OFF motor. In other words, if we have at least one of the classes with a probability higher than, say, 80%, we are close enough to the blobs of training data.
2. If the probability of ON and OFF is somewhere around 50% then we very likely have a defective motor.

One can choose the probability threshold to his liking (be it 80%, 70% or 90%), as a way to adjust for false positives or false negatives.

### Issues with this solution

• The assumption that a defective motor will have vibrations between the ON and OFF classes. i.e. if a defect makes the motor vibrate more than the normal ON it will be undetectable with this solution.
• Assumes linearity and same scale of distance from ON and from OFF for defective motors. i.e. classes are of the same size and equally distributed. Such solution will check distances from each class using the same linear scale (the threshold), it may not detect defects that happen close to the smaller class.

# Use Novelty Detection

You do have two known classes: ON and OFF. But what you are trying to find is whether a motor is defective or not, where we understand non-defective as the fact that the motor is very similar to most ON motors or very similar to most OFF motors.

Forget about the ON and OFF classes and dump both sets of samples of non-defective motors as one single (non-defective) class and into a novelty detection algorithm. In other words, do not differentiate between ON and OFF motors. Say that both cases (ON and OFF) are OK, just anything that is quite far away from any known cases is not-OK. This will look as a bunch of blobs in space and anything far away from these blobs is a defective motor.

I'll suggest starting with an One Class SVM because it has a parameter called nu= which is quite easy to interpret. It varies between 0 and 1 and the closest to zero it is the more the algorithm is permissive of anomalies (defects) away from the blobs.

Note that we have exactly the same points on this picture as in the previous one. The only difference is that this time all of them are red.

This solution will cope with defects that are anywhere, not only in between the states of ON and OFF.

### Issues with this solution

• On Class SVM does not have a .predict_proba. To get an estimate of how far away from a blob an anomaly marked as an anomaly is, one must evaluate the .decision_function of the model and keep track of the distances oneself.
• Even if the classes ON and OFF are well separated one will have false positives and false negatives. A very low nu= parameter still cannot go beyond the blobs. Using an Isolation Forest may allow for better tuning but the decision function would be even more complex to get distances from. Another option is to add marginal versions of ON and OFF to the training data but finding those may not be easy.

P.S. If one needs to do both classify ON and OFF AND classify defective and non-defective motors, then one should build two separate models.

• Unfortunately I cannot do this assumption _Defective motors will produce less vibrations than motors ON, but more vibrations than motors OFF _. In fact is almost always the opposite. However the other solution can be really helpful, using 2 models as you said it might work for the inteire solution. I have just two doubts, I have vibration from 3 axis + 1 temp, can I still use Novelty Detection (isnot a 2 d plan)? Do you have any idea where I can study the .decision_function for Novel. detec.? Thaaaanks a lot!! – André Braga Jun 17 '19 at 19:51