unsupervised anomaly detection on sparse data

Question

Given that I have a very sparse data matrix with continuous features, like this dataframe for example

 Feature_A  Feature_B  Feature_C....Feature_Z  
 0.3            0       0.1            0
 0.5            0.5     0              0
 0              0       1.0            0
 1.0            0       0              0  
 0.7            0       0              0
 1.0            0       0              0
 0.1            0       0.22          0.43

what is the best way to perform unsupervised anomaly detection on this kind of data? my initial idea was to perform some kind of dimensionality reduction first (e.g SVD or NMF) then do a simple anomaly detection technique on the resultant dense matrix (e.g Isolation Forest) but I'm not sure this is the best way to go.

Answer 1

1- You better start with Isolation forest Isolation Forest This is a very simple algorithm where you can control the contamination rate of your data.

2- For visualization you can plot the anomalous points in red, and you can distinguish them using the Isolation Forest predict(X) function that returns -1 for outliers and 1 for the rest.

3- You can use LSTM by comparing the predicted values of your model, with the real values of your dataset, then use a KPI ( if/else depending on the max value of the difference between the predicted values and the real ones), in simple words you will need to define your own contamination rate.

4- For visualization with LSTM you can use the same function described above with Isolation Forest, but instead of the predict function that return [-1,1], you will use the function described in step 4.