Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
1
vote
1answer
15 views

SPC vs Autoencoders in anomaly detection

Considering the usage of Autoencoders in anomaly detection of time-series data, why SPCs (control charts) have lost their charm? Are there any advantages with Autoencoders and disadvantages with SPCs?
0
votes
0answers
16 views

Custom Decision Function for Custom Outlier Detection Algorithm

I have built a custom algorithm for semi-supervised anomaly detection and here is my output example as following with probability threshold set to 0.05 and 1 = outlier, 0 = inlier: ...
0
votes
0answers
22 views

Time Series Anomaly Detection [closed]

I have been given an anomaly detection problem as follows. I want to find the anomaly score in my dataset given that it has 8 features (coming from 8 different sensors). What are the steps and best ...
0
votes
0answers
12 views

How do I evaluate a K-Means unsupervised anomaly detection approach?

how do I evaluate K-means clustering anomaly detection method as there is no labelled data of anomaly class. To find the cluster (K), I have used the silhouette score from Scikit learn library. Scikit ...
0
votes
0answers
14 views

is it beneficial to use high-order n-grams as feature vectors for web anomaly detection?

i am studying about the use of n-gram models to classify web attacks based on several parameters like, requested resources, query parameters and attributes, characters distribution and so on. Most ...
0
votes
0answers
16 views

Clustering a dataset and creating a model per each cluster

I was wondering if it makes sense to cluster a dataset to find closely related data points and train a binary classification model for each of this clusters as they would be minidatasets. I'll ...
0
votes
2answers
30 views

Percentile as a threshold for Anomaly Detection?

I'm following this article about Unsupervised Anomaly Detection Algorithms. In this article, a threshold value is calculated using the scipy score percentile method to determine whether the point is ...
-1
votes
2answers
39 views

Plotting ROC & AUC for SVM algorithm

Towards , the end of my program, I have the following code. model = svm.OneClassSVM(nu=nu, kernel='rbf', gamma=0.00001) model.fit(train_data) Output ...
2
votes
2answers
53 views

Many separation line using RBF kernel in SVM

Below is my code, it take a range of a number, creates a new column label that contains either -1 or 1. In case the number is ...
3
votes
3answers
39 views

K-Means anomaly detection not clustering anomalies

K-means anomaly detection scatter plot The following code, takes a single column from a dataset and then adds 50 anomalies to the dataset that is quite bigger than the maximum values of the dataset. ...
0
votes
0answers
5 views

How to model anomaly data using Gaussian distribution assuming variables are dependent? (In Python)

I have some data which contains anomalies as well. I want to model data using Gaussian distribution assuming variables are dependent in Python. If covariance matrix is the answer, then how do we ...
-1
votes
0answers
12 views

Timeseries multivariate anamoly detection using neural network

I am planning to take up this project to identify service degradation for users. I want to build a system that identifies sudden service degradation for subscribers based on about 50 variables. I ...
0
votes
0answers
11 views

How to set anomaly threshold depending of predictive model accuracy

Say I have a variable with a standard deviation STD I have a predictive model to predict variable. The model accuracy is 80% An anomaly is raised if difference (predicted_value - actual value) > ...
1
vote
0answers
16 views

Anomaly Detection Methods for Clean Training Data

The goal is to find anomalies in my dataset, univariant anomaly detection which basically means looking for anomalies only in one column. Up until this point I have tried DBSCAN; Isolation Forest and ...
2
votes
1answer
34 views

Adding anomalies to the Dataset

Recently I have been trying different Scikit-Learn anomaly detection clustering methods, like DBSCAN Isolation Forest. Based on how many training data I use, how I tweak on the algorithms ...
0
votes
0answers
23 views

Online courses for Anomaly Detection

As the title say I have been looking for some online courses that would teach me about anomaly detection using Unsupervised Machine Learning. I want to focus only on Scikit-Learn and not go deeper ...
2
votes
1answer
36 views

Outlier Detection using K-Means using one column

I have done and read a csv file and then plotted the values of a single column using K-means ...
0
votes
0answers
14 views

box plot and the anomaly detection?

i'm doing a classification problem with 50000 rows × 5000 columns of dataset. calssification label is 300 labels 1. This is the few example of box plot of some feature. what can I inference from ...
0
votes
0answers
13 views

PyOD algorithms that support datasets with empty cells

The idea is to find outliers in large datasets. What am I going to use to detect these outliers? I am going to use a library called PyOD for the detection of anomalies which was developed by Yue ...
1
vote
1answer
23 views

Effects that Empty cells have in Unsupervised Machine Learning

I have a dataframe in a csv file that I would like to perform different unsupervised machine learning algorithms on. The file itself has some cells that are not ...
0
votes
1answer
23 views

single v/s multiple user machine learning model

What are the merits and demerits of training machine learning models(One-class svm, isolation forest, DBSCAN) for anomaly detection on single-user data set and multiple user data set.? keep in mind ...
0
votes
0answers
21 views

Are there any methods to detect whole multivariate time-series as anomalous from a set of multivariate time-series?

Consider a scenario with Dataset D as {T1, T2, ..., Tn} and Ti is a multivariate time-series of length mi as {X1, X2, ..., Xmi}. Here each record of the time-series Xi is a vector of attribute values {...
2
votes
0answers
24 views

Machine Learning Techinques that Automate Fast Fourier Transform

I have a 40k Hz time-series data of vibration, which is used to predict equipment failure. The goal here is to make a system that predicts it automatically. I am thinking of a couple of ways but not ...
1
vote
2answers
79 views

Anomaly Detection/Novelty detection

I have a data-set that has over 6 million normal data and around 50 anomaly data.Those anomaly data is identified by manually(monitoring the user`s activity over camera and identify). I need to ...
2
votes
1answer
34 views

Should I transform my feature into normal distrubition before Isolation Forest

I have a anomaly detection problem and my features are following exponential distrubition. Should I first transform my features into normal distrubition before feed into isolation forest?
1
vote
0answers
15 views

Built strong base for Unsupervised Learning [closed]

I’m am new into machine learning, recently I have put a task upon my shoulders to Detect Outliers in Dataset. The anomaly detection should be done using Unsupervised learning and preferably use ...
0
votes
0answers
40 views

Autoencoder for anomaly detection, output layer activation function

I am building an Autoencoder to detect anomalies. I have mixed data, i.e continuous and categorical. I have one-hot encoded the categorical data. Scaled the data with a MinMax scaler. To determine if ...
1
vote
1answer
42 views

Autoencoder anamoly detection

I recently learnt about the anamoly detection using autoencoders(specifically denoisinng autoencoders).To train the autoencoders do we need a data having some pattern? or is there some way to ...
0
votes
0answers
34 views

Learning Process in Machine Learning

I want to use Unsupervised Leaning to detect Anomalies within a huge Csv file (consisting of headers that are named and thousands of rows belonging to them) Here on this link I have read about the ...
3
votes
1answer
102 views

Best anomaly detection algorithm based on two conditions

Choosing the right anomaly detection algorithm seems quite hard at the moment. It might be because I am bombarded with so many alternatives likes clustering, K-Means DBSCAN and so many others. On my ...
3
votes
4answers
92 views

Machine Learning methods for finding outliers

I have a csv files of thousands of lines, the data is put down into a Dataframe of columns. Some of the column have text information while others might have numbers. I want to detect anomalies or ...
1
vote
0answers
19 views

How to handle data with dependency on two different dates

I am currently dealing with a dataset that contains multiple date-time fields: "buy-date" and "receive-date" which both have an effect on the prices and amount of offers made. One example could be: <...
3
votes
0answers
15 views

High dimensional data stream summarization and processing

Can anyone recommend a method for summarizing and processing high dimensional data streams efficiently and effectively for anomaly detection? In fact, I investigated the different methods for data ...
0
votes
1answer
23 views

looking for approaches to detecting outliers in individuals unequal sequential time series

I am looking for approaches related to outlier detection in time series. Example: A person visits hospital overtime on multiple bases and there are some measurements made (bmi, blood_pressure, ...
0
votes
0answers
30 views

I am trying to implement Isolation forest for anomaly detection but I am not able to understand and visualise the decision function

I am trying to implement Isolation forest for anomaly detection but I am not able to understand the scores. Also, I want to plot and visualise the decision function that my Isolation forest is ...
0
votes
1answer
17 views

Are cluster feature and micro-cluster good summury statics for outlier detection in high dimensional data streams?

I'm dealing with outlier detection in data streams. I'm looking for a way to summarize my data and obtain important statistics such as means and variance, etc. I want to know if the cluster features ...
0
votes
0answers
21 views

Which features are selected in isolation forest h2o?

As a method for anomaly detection, I'm looking at isolation forest in the H2O implementation in Python. I'd like to identify the features corresponding to a certain data point in a tree. I can see ...
1
vote
1answer
68 views

Anomaly detection - relation between thresholds and anomalies

I'm developing an anomaly detection program in Python. Main idea is to create a new LSTM model every day, training it with the previous 7 days and predict the next day. Then, using thresholds, find ...
2
votes
1answer
36 views

ML Approach for Graph Anomaly Detection

Very new to ML. I am trying to create an anomaly detector. I have thousands of graphs like the one I have attached. I am interested in the pink line. If the pink line's behavior changes drastically ...
2
votes
1answer
9 views

Parameter Adjustment based only on tagged predictions

not sure that this is the best place to post this but if not, please let me know if there is a better stack community. I have an anomaly detection method which has some parameters. I have some data ...
0
votes
1answer
19 views

Testing if a sample fits into an existing cluster

I have a sample of data I'd like to create a model from, which would create N clusters. After the fitting to clusters, I'd like to test various samples against the existing clusters, seeing if the ...
0
votes
0answers
21 views

Can I use/modify an Autoencoder to handle missing data?

I am about to implement an Autoencoder to detect anomalies. Therefore, e.g., in my test set, there is a situation where the data stream broke for some days. This results in a lack of data and should ...
0
votes
1answer
297 views

Sampling labeled data for anomaly detection

I'm currently working on a project that requires the use of unsupervised anomaly detection, but I'm unable to find a relavent data set, so I'm considering the following option: Assuming I have a data ...
1
vote
1answer
73 views

Anomaly detection thresholds issue

I'm working on an anomaly detection development in Python. More in details, I need to analysed timeseries in order to check if anomalies are present. An anomalous value is typically a peak, so a ...
2
votes
0answers
15 views

Temporal outlier Analysis on sensor data

I am working to find anomaly/outliers in sensor data using unsupervised machine learning (without training dataset). I have around 20000 samples taken per minute of various sensors. I just need to ...
0
votes
0answers
10 views

Perform unsupervised anomaly identification with causation in Python?

I have a time series data, which contains information from various sensors measured at every 20 minutes interval. I would like to use information from all these sensors as features to a Deep Learning/...
0
votes
1answer
166 views

Explainable anomaly detection

There are plenty of working for explaining prediction in supervised learning (e.g. SHAP values, LIME). What about for anomaly detection in unsupervised learning? Is there any model for which there ...
0
votes
0answers
17 views

VAE latent space dimension with Weibull PDFs as input data

I am currently using a Variational Autoencoder to reconstruct PDFs of Weibull distributions with varying parameters (sampled uniformly from a given parameter span). The PDFs are generated by sampling ...
0
votes
0answers
11 views

LSTM based anomaly detection scheme too closely tracking long spans of anomalous points

I've built a time series anomaly detection process that accurately predicts the value at the next interval. However, when there are dozens of anomalous events in a row, the model starts to "catch up" ...
1
vote
1answer
28 views

Functions in scikit that detect outliers automatically?

I know a way to visualize outliers is to make a box plot, but wanted to know if scikit had any quick ways to detect outliers for each variable?

1
2 3 4 5