Anomaly detection for varying dictionary

Question

I want to detect the anomaly in the processes taking up the most CPU percent. I receive the data as a time series of dictionary values like so:

    time                         process_most_cpu                                                             cpu%
0   2022-02-22 21:04:57.021740  {'chromium-browse': 38.70,'python': 32.00,'mutter': 2.90,'python3': 1.60}    26.10
1   2022-02-22 21:05:32.836466  {'chromium-browse': 25.70,'mutter': 2.90,'python3': 1.60}                    34.50
2   2022-02-22 21:05:55.558390  {'chromium-browse': 21.70,'python': 5.80,'mutter': 2.90,'python3': 1.50}      5.70
3   2022-02-22 21:07:01.069036  {'pip': 37.90,'chromium-browse': 19.30,'mutter': 2.90,'python3': 1.50}       11.70

I'm not sure how to detect the anomaly here as the processes keep on changing. Feature extraction methods such as one-hot encoding don't seem to work in this particular case due to the varying dictionary keys.

Any advice would be very much appreciated!

Answer 1

Create a ref dictionary in which the time-series of each component is appended to a list. You will end up with multiple time-series and just simply say if a time-series deviates dramatically from its moving average, it is an anomaly. This means, I would not go with feature extraction as your input will be literally some temporal info. This is a classic time-series analysis problem and better to look at it that way.

Please note that the anomaly detection here is pretty basic and based on simple statistics. You can go much more complex e.g. train a prediction model for each time-series and output anomaly if prediction and actual value at a time are very (based on a user threshold) different.