Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0
votes
0answers
32 views

Anomaly Detection over multivariate data containing Nominal and numerical predictors

I am trying to implement Anomaly Detection over a multivariate dataset having nominal and numerical predictors. Dataset has following pattern: If we consider the below sample records, category_id, ...
-1
votes
1answer
23 views

Does Anomaly Detection Algorithm works when the features are not correlated?

I am working on an Anomaly Detection Problem and the algorithm I used is an Autoencoder Multivariate Gaussian. The problem with my data is that it is unlabeled and not correlated. For example, let's ...
1
vote
1answer
24 views

Interpretation of scikit-learn one class svm scores

How can I interpret the scores generated by the function score_samples(X) from a scikit-learn OneClassSVM model? Is there a way ...
0
votes
1answer
27 views

Semi-supervised anomaly detection

I am currently exploring anomaly detection methods for my work and, basically I have gone through Local Oulier Factor and Isolation Forests, both unsupervised methods. Now, the thing is, there might ...
1
vote
2answers
26 views

Identify the parameter causing the anomaly in a multivariate dataset

I have a payment transaction dataset with a large number of predictor variables. I am trying to build a model for anomaly detection and I have evaluated various algorithms/approaches for the same like ...
2
votes
0answers
27 views

What's the best way to validate a rare event detection model during training?

When training a deep model for rare event detection (e.g. sound of an alarm in a home device audio stream), is it best to use a balanced validation set (50% alarm, 50% normal) to determine early ...
1
vote
0answers
63 views

Hyperparameter tuning one-class svm

I have a problem where I am trying to apply a one-class svm to detect outliers. I am training on a dataset of true cases using a one-class radial svm and then predicting for both false and true cases. ...
2
votes
0answers
25 views

anomaly detection in vehicle sensor data

I am currently diving deeper into understanding more about anomaly detection in regards to vehicle's data generated by sensors. It seems like there is no proper book or article that goes deeper into ...
0
votes
0answers
20 views

Which graph to choose to plot anomaly detection

I'm creating tool to detect anomalies in syslog messages. I'm parsing syslogs into Bag of Words of 200 features. This BoW is forwarded into machine learning model based on selection. I got 12 models - ...
0
votes
1answer
43 views

Can we use one hot encoding instead of for loops?

I have an anomaly detection model, that I run per store with a bunch of features. I intend to run this code, everyday, per store. Now, lets say I have 8000 stores, I would imagine, I should write a ...
2
votes
0answers
17 views

Types of artificial anomalies

I am working on some algorithms for anomaly detection The dataset is clean our anomalies so I want to add some artificial anomalies. I have added some anomalies. I get the maximum value of the ...
0
votes
0answers
20 views

Isolation forest challenge with contamination factor

I do isolation forest with time series data for anomaly detection. Its unsupervised model which detects anomaly with past 2 weeks data on todays data the window moves forward everyday Due to auto ...
0
votes
2answers
24 views

What does the classification report interpret? Class 1 indicates abnormal data

How to interpret the report and How is precision, recall values are calculated for individual class labels. What is the significance of macro avg ? Does this report signify a good predictions by the ...
1
vote
0answers
23 views

Classifier for DBSCAN [closed]

I have written a code that uses DBSCAN and tries to find the most appropriate eps for my dataset, trying to include most of the data inside a cluster. The problem is that DBSCAN is not a classifier ...
3
votes
1answer
44 views

Outlier/Anomaly Detection History

I have been reading about different methods of anomaly detection, their structure and the way they work. Recently I have been trying to find some scholar articles, writings or books where I can learn ...
0
votes
0answers
8 views

Anomalies timeseries: discard feature vectors or samples?

I'm implementing a classifier which classifies based on a time-series. This time-series is made out of a 50Hz sensor and measures three items (x, y, z). Yet sometimes there's an unusually high value, ...
0
votes
0answers
17 views

Is there any know algorithm or technique to detect IOT clock anomaly

I have some IOT machine in place and a user have to present a NFC card on it when using one of this machine. When the machine is detecting the card it send an event to a backoffice system with the ...
0
votes
0answers
10 views

Reporting false alarm rate and detection rate in shilling attack detection without given any labels

I have a dataset of ratings given to movies by users. I've applied the method mentioned in this paper to detect fake votes(I've used $H(X)$ and $M(X)$ measures). But the dataset I'm using doesn't have ...
1
vote
1answer
23 views

SPC vs Autoencoders in anomaly detection

Considering the usage of Autoencoders in anomaly detection of time-series data, why SPCs (control charts) have lost their charm? Are there any advantages with Autoencoders and disadvantages with SPCs?
0
votes
0answers
21 views

Custom Decision Function for Custom Outlier Detection Algorithm

I have built a custom algorithm for semi-supervised anomaly detection and here is my output example as following with probability threshold set to 0.05 and 1 = outlier, 0 = inlier: ...
0
votes
0answers
13 views

How do I evaluate a K-Means unsupervised anomaly detection approach?

how do I evaluate K-means clustering anomaly detection method as there is no labelled data of anomaly class. To find the cluster (K), I have used the silhouette score from Scikit learn library. Scikit ...
0
votes
0answers
15 views

is it beneficial to use high-order n-grams as feature vectors for web anomaly detection?

i am studying about the use of n-gram models to classify web attacks based on several parameters like, requested resources, query parameters and attributes, characters distribution and so on. Most ...
0
votes
0answers
18 views

Clustering a dataset and creating a model per each cluster

I was wondering if it makes sense to cluster a dataset to find closely related data points and train a binary classification model for each of this clusters as they would be minidatasets. I'll ...
0
votes
2answers
65 views

Percentile as a threshold for Anomaly Detection?

I'm following this article about Unsupervised Anomaly Detection Algorithms. In this article, a threshold value is calculated using the scipy score percentile method to determine whether the point is ...
-1
votes
2answers
71 views

Plotting ROC & AUC for SVM algorithm

Towards , the end of my program, I have the following code. model = svm.OneClassSVM(nu=nu, kernel='rbf', gamma=0.00001) model.fit(train_data) Output ...
2
votes
2answers
61 views

Many separation line using RBF kernel in SVM

Below is my code, it take a range of a number, creates a new column label that contains either -1 or 1. In case the number is ...
2
votes
3answers
52 views

K-Means anomaly detection not clustering anomalies

K-means anomaly detection scatter plot The following code, takes a single column from a dataset and then adds 50 anomalies to the dataset that is quite bigger than the maximum values of the dataset. ...
0
votes
1answer
13 views

How to model anomaly data using Gaussian distribution assuming variables are dependent? (In Python)

I have some data which contains anomalies as well. I want to model data using Gaussian distribution assuming variables are dependent in Python. How can I model this? Should I use the PDF formula as ...
0
votes
0answers
12 views

How to set anomaly threshold depending of predictive model accuracy

Say I have a variable with a standard deviation STD I have a predictive model to predict variable. The model accuracy is 80% An anomaly is raised if difference (predicted_value - actual value) > ...
1
vote
0answers
18 views

Anomaly Detection Methods for Clean Training Data

The goal is to find anomalies in my dataset, univariant anomaly detection which basically means looking for anomalies only in one column. Up until this point I have tried DBSCAN; Isolation Forest and ...
2
votes
1answer
39 views

Adding anomalies to the Dataset

Recently I have been trying different Scikit-Learn anomaly detection clustering methods, like DBSCAN Isolation Forest. Based on how many training data I use, how I tweak on the algorithms ...
0
votes
0answers
35 views

Online courses for Anomaly Detection

As the title say I have been looking for some online courses that would teach me about anomaly detection using Unsupervised Machine Learning. I want to focus only on Scikit-Learn and not go deeper ...
2
votes
1answer
48 views

Outlier Detection using K-Means using one column

I have done and read a csv file and then plotted the values of a single column using K-means ...
0
votes
0answers
19 views

box plot and the anomaly detection?

i'm doing a classification problem with 50000 rows × 5000 columns of dataset. calssification label is 300 labels 1. This is the few example of box plot of some feature. what can I inference from ...
0
votes
0answers
18 views

PyOD algorithms that support datasets with empty cells

The idea is to find outliers in large datasets. What am I going to use to detect these outliers? I am going to use a library called PyOD for the detection of anomalies which was developed by Yue ...
1
vote
1answer
33 views

Effects that Empty cells have in Unsupervised Machine Learning

I have a dataframe in a csv file that I would like to perform different unsupervised machine learning algorithms on. The file itself has some cells that are not ...
0
votes
1answer
33 views

single v/s multiple user machine learning model

What are the merits and demerits of training machine learning models(One-class svm, isolation forest, DBSCAN) for anomaly detection on single-user data set and multiple user data set.? keep in mind ...
0
votes
0answers
22 views

Are there any methods to detect whole multivariate time-series as anomalous from a set of multivariate time-series?

Consider a scenario with Dataset D as {T1, T2, ..., Tn} and Ti is a multivariate time-series of length mi as {X1, X2, ..., Xmi}. Here each record of the time-series Xi is a vector of attribute values {...
2
votes
0answers
30 views

Machine Learning Techinques that Automate Fast Fourier Transform

I have a 40k Hz time-series data of vibration, which is used to predict equipment failure. The goal here is to make a system that predicts it automatically. I am thinking of a couple of ways but not ...
0
votes
1answer
138 views

Anomaly Detection/Novelty detection

I have a data-set that has over 6 million normal data and around 50 anomaly data. Those anomaly data is identified manually (by monitoring the user`s activity over camera and identify). I need to ...
2
votes
1answer
36 views

Should I transform my feature into normal distrubition before Isolation Forest

I have a anomaly detection problem and my features are following exponential distrubition. Should I first transform my features into normal distrubition before feed into isolation forest?
1
vote
0answers
15 views

Built strong base for Unsupervised Learning [closed]

I’m am new into machine learning, recently I have put a task upon my shoulders to Detect Outliers in Dataset. The anomaly detection should be done using Unsupervised learning and preferably use ...
0
votes
0answers
59 views

Autoencoder for anomaly detection, output layer activation function

I am building an Autoencoder to detect anomalies. I have mixed data, i.e continuous and categorical. I have one-hot encoded the categorical data. Scaled the data with a MinMax scaler. To determine if ...
1
vote
1answer
44 views

Autoencoder anamoly detection

I recently learnt about the anamoly detection using autoencoders(specifically denoisinng autoencoders).To train the autoencoders do we need a data having some pattern? or is there some way to ...
0
votes
0answers
34 views

Learning Process in Machine Learning

I want to use Unsupervised Leaning to detect Anomalies within a huge Csv file (consisting of headers that are named and thousands of rows belonging to them) Here on this link I have read about the ...
3
votes
1answer
166 views

Best anomaly detection algorithm based on two conditions

Choosing the right anomaly detection algorithm seems quite hard at the moment. It might be because I am bombarded with so many alternatives likes clustering, K-Means DBSCAN and so many others. On my ...
3
votes
4answers
136 views

Machine Learning methods for finding outliers

I have a csv files of thousands of lines, the data is put down into a Dataframe of columns. Some of the column have text information while others might have numbers. I want to detect anomalies or ...
1
vote
0answers
21 views

How to handle data with dependency on two different dates

I am currently dealing with a dataset that contains multiple date-time fields: "buy-date" and "receive-date" which both have an effect on the prices and amount of offers made. One example could be: <...
3
votes
0answers
19 views

High dimensional data stream summarization and processing

Can anyone recommend a method for summarizing and processing high dimensional data streams efficiently and effectively for anomaly detection? In fact, I investigated the different methods for data ...
0
votes
1answer
26 views

looking for approaches to detecting outliers in individuals unequal sequential time series

I am looking for approaches related to outlier detection in time series. Example: A person visits hospital overtime on multiple bases and there are some measurements made (bmi, blood_pressure, ...

1
2 3 4 5