Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0
votes
1answer
22 views

Decision trees for anomaly detection

Problem From what I understand, a common method in anomaly detection consists in building a predictive model trained on non-anomalous training data, and perform anomaly detection using the error of ...
0
votes
1answer
17 views

Detecting spikes from voltage channels (in python) [closed]

So I am pretty new to data science in general, and I've been tasked with finding the "spikes" in data, or every time the voltage (the data in working with) crosses a specific "threshold&...
0
votes
0answers
11 views

Analysing wrongly predicted labels and pick most significant

I try to do behavior analysis of network systems based on network flow. I have a model that predicts if a certain flow pattern is a Windows Server, LX Server or Windows Client. (Highly simplified) I ...
2
votes
2answers
33 views

How to perform Anomaly Detection on a force profile?

I have a set of force profiles of an industrial machine. I'm trying to develop an algorithm that tries to understand when a new profile is "anomalous" with respect to the ones in "...
0
votes
1answer
22 views

unsupervised anomaly detection on sparse data

Given that I have a very sparse data matrix with continuous features, like this dataframe for example ...
0
votes
0answers
19 views

Anomaly detection model on 1D data but not Gaussian distribution

I have a scenario that I have one-dimensional data and the distribution of the data is not Gaussian. (more like a bimodal distribution). I want to find an anomaly detection model that could predict ...
0
votes
0answers
25 views

How to find a 'similarity' measure between 2 pandas dataframes?

I have 2 pandas dataframes: one big (300.000+ rows) and one little (50- rows) with the same columns. Assuming that the entries of the little one form a cluster I want to identify the entries of the ...
1
vote
1answer
49 views

K-Prototype for anomaly detection

I have logs of the form (e.g. from a gym login).. the representational case is so: UserName, Login time, timeSpend_on_weights, time_spent_on_elliptical ...
1
vote
0answers
17 views

Explanation of Excess Mass(EM)

I was researching on evaluation metrics to understand the performance of unsupervised anomaly detection algorithms and I came across this paper The author suggests that EM and MV based numerical ...
10
votes
1answer
563 views

Learning with Positive labels only

I have ~7 million rows of customer data (~500 sparse attributes) A million out of them have opted in to a new service. How do I use this signal to predict which of the remaining customers are likely ...
1
vote
1answer
21 views

Is window based sequencing a good idea to obtain more training data for LSTMs?

I am trying to do an unsupervised autoencoder based outlier detection for time series using LSTMs. Here, there are multiple time series, and an entire series is to be considered as an outlier. However,...
2
votes
2answers
27 views

Log analysis dataset with labeled cybersecurity issues

I am seeking to find a dataset with log files that have labeled cybersecurity issues. As I am trying to build a cybersecurity log analysis model there is no preference on the type of the log, but ...
0
votes
0answers
15 views

Can I treat concept drift detection as a binary clarification problem?

Assume I have the ground truth labels for both non-drited and drifted samples, can I treat concept drift detection as a binary classication problem (one class non-drifted one class drifted)? If not ...
0
votes
2answers
28 views

Looking for spatial clusters and anomalies. Is DBSCAN the right tool?

I have a regular 2D grid of data points (X, Y) with each point having a value. I'd like to identify clusters and then anomalies that don't belong to those clusters. I'm trying to understand the best ...
2
votes
1answer
21 views

Word representation that gives more weight to terms frequent in corpus?

The tf-idf discounts the words that appear in a lot of documents in the corpus. I am constructing an anomaly detection text classification algorithm that is trained only on valid documents. Later I ...
0
votes
1answer
32 views

Algorithms for Anomaly Detection of Event Sequence Data [Python/R]

I am building an anomaly detection system of event sequence data (transactions). For each timestep, a transaction can be in any of 76 different stages. My dataset is therefore a 3D array of size(m,t,N)...
0
votes
0answers
16 views

Identifying anomalies in spatial (latitude, longitude, time) data? [duplicate]

I'll start off by prefacing this with the fact that I'm not even sure if I'm asking the right question and what I'm really looking for is some guidance to dig in the right direction. I have a set of ...
0
votes
1answer
33 views

Incorrect multi-variate anomaly detection - Isolation Forest Python

My data looks like below. it has 333 rows and 2 columns. Clearly the first row is anomaly. ndf: ...
0
votes
0answers
11 views

Is there any inductive Graph Variational Auto Encoder?

I have been reading about how we can model a Variational AutoEncoder (VAE) into a Graph Variational AutoEncoder (GVAE) where the decoder reconstructs the adjacency matrix. I presume that this approach ...
0
votes
2answers
42 views

Cross-Validation in Anomaly Detection with Labelled Data

I am working on a project where I train anomaly detection algorithms Isolation Forest and Auto-Encoder. My data is labelled so I have the ground truth but the nature of the problem requires ...
1
vote
0answers
20 views

Multivariate Gaussian distribution - Covariance vs linear dependence

From prof. Andrew Ng's Multivariate Gaussian distribution lecture, covariance measures linear dependency between features, in which case we might use Multivariate Gaussian distribution with covariance ...
0
votes
0answers
50 views

Anomaly Detection over multivariate data containing Nominal and numerical predictors

I am trying to implement Anomaly Detection over a multivariate dataset having nominal and numerical predictors. Dataset has following pattern: If we consider the below sample records, category_id, ...
-1
votes
1answer
29 views

Does Anomaly Detection Algorithm works when the features are not correlated?

I am working on an Anomaly Detection Problem and the algorithm I used is an Autoencoder Multivariate Gaussian. The problem with my data is that it is unlabeled and not correlated. For example, let's ...
2
votes
1answer
213 views

Interpretation of scikit-learn one class svm scores

How can I interpret the scores generated by the function score_samples(X) from a scikit-learn OneClassSVM model? Is there a way ...
0
votes
1answer
48 views

Semi-supervised anomaly detection

I am currently exploring anomaly detection methods for my work and, basically I have gone through Local Oulier Factor and Isolation Forests, both unsupervised methods. Now, the thing is, there might ...
1
vote
2answers
36 views

Identify the parameter causing the anomaly in a multivariate dataset

I have a payment transaction dataset with a large number of predictor variables. I am trying to build a model for anomaly detection and I have evaluated various algorithms/approaches for the same like ...
2
votes
0answers
31 views

What's the best way to validate a rare event detection model during training?

When training a deep model for rare event detection (e.g. sound of an alarm in a home device audio stream), is it best to use a balanced validation set (50% alarm, 50% normal) to determine early ...
1
vote
0answers
104 views

Hyperparameter tuning one-class svm

I have a problem where I am trying to apply a one-class svm to detect outliers. I am training on a dataset of true cases using a one-class radial svm and then predicting for both false and true cases. ...
2
votes
0answers
31 views

anomaly detection in vehicle sensor data

I am currently diving deeper into understanding more about anomaly detection in regards to vehicle's data generated by sensors. It seems like there is no proper book or article that goes deeper into ...
0
votes
0answers
24 views

Which graph to choose to plot anomaly detection

I'm creating tool to detect anomalies in syslog messages. I'm parsing syslogs into Bag of Words of 200 features. This BoW is forwarded into machine learning model based on selection. I got 12 models - ...
0
votes
1answer
54 views

Can we use one hot encoding instead of for loops?

I have an anomaly detection model, that I run per store with a bunch of features. I intend to run this code, everyday, per store. Now, lets say I have 8000 stores, I would imagine, I should write a ...
2
votes
0answers
19 views

Types of artificial anomalies

I am working on some algorithms for anomaly detection The dataset is clean our anomalies so I want to add some artificial anomalies. I have added some anomalies. I get the maximum value of the ...
0
votes
0answers
86 views

Isolation forest challenge with contamination factor

I do isolation forest with time series data for anomaly detection. Its unsupervised model which detects anomaly with past 2 weeks data on todays data the window moves forward everyday Due to auto ...
0
votes
2answers
273 views

What does the classification report interpret? Class 1 indicates abnormal data

How to interpret the report and How is precision, recall values are calculated for individual class labels. What is the significance of macro avg ? Does this report signify a good predictions by the ...
1
vote
0answers
30 views

Classifier for DBSCAN [closed]

I have written a code that uses DBSCAN and tries to find the most appropriate eps for my dataset, trying to include most of the data inside a cluster. The problem is that DBSCAN is not a classifier ...
3
votes
1answer
51 views

Outlier/Anomaly Detection History

I have been reading about different methods of anomaly detection, their structure and the way they work. Recently I have been trying to find some scholar articles, writings or books where I can learn ...
0
votes
0answers
8 views

Anomalies timeseries: discard feature vectors or samples?

I'm implementing a classifier which classifies based on a time-series. This time-series is made out of a 50Hz sensor and measures three items (x, y, z). Yet sometimes there's an unusually high value, ...
0
votes
0answers
12 views

Reporting false alarm rate and detection rate in shilling attack detection without given any labels

I have a dataset of ratings given to movies by users. I've applied the method mentioned in this paper to detect fake votes(I've used $H(X)$ and $M(X)$ measures). But the dataset I'm using doesn't have ...
1
vote
1answer
44 views

SPC vs Autoencoders in anomaly detection

Considering the usage of Autoencoders in anomaly detection of time-series data, why SPCs (control charts) have lost their charm? Are there any advantages with Autoencoders and disadvantages with SPCs?
0
votes
0answers
24 views

Custom Decision Function for Custom Outlier Detection Algorithm

I have built a custom algorithm for semi-supervised anomaly detection and here is my output example as following with probability threshold set to 0.05 and 1 = outlier, 0 = inlier: ...
0
votes
0answers
17 views

How do I evaluate a K-Means unsupervised anomaly detection approach?

how do I evaluate K-means clustering anomaly detection method as there is no labelled data of anomaly class. To find the cluster (K), I have used the silhouette score from Scikit learn library. Scikit ...
2
votes
0answers
54 views

How Yelp System Detects Paid Reviews

I am wondering how the yelp spam detection system detects paid reviews? By paid, I mean the following scenarios: I as a business owner pay people to write positive comments and give me a good rate I ...
0
votes
0answers
22 views

is it beneficial to use high-order n-grams as feature vectors for web anomaly detection?

i am studying about the use of n-gram models to classify web attacks based on several parameters like, requested resources, query parameters and attributes, characters distribution and so on. Most ...
0
votes
0answers
19 views

Clustering a dataset and creating a model per each cluster

I was wondering if it makes sense to cluster a dataset to find closely related data points and train a binary classification model for each of this clusters as they would be minidatasets. I'll ...
0
votes
2answers
181 views

Percentile as a threshold for Anomaly Detection?

I'm following this article about Unsupervised Anomaly Detection Algorithms. In this article, a threshold value is calculated using the scipy score percentile method to determine whether the point is ...
-1
votes
2answers
326 views

Plotting ROC & AUC for SVM algorithm

Towards , the end of my program, I have the following code. model = svm.OneClassSVM(nu=nu, kernel='rbf', gamma=0.00001) model.fit(train_data) Output ...
2
votes
2answers
69 views

Many separation line using RBF kernel in SVM

Below is my code, it take a range of a number, creates a new column label that contains either -1 or 1. In case the number is ...
2
votes
3answers
63 views

K-Means anomaly detection not clustering anomalies

K-means anomaly detection scatter plot The following code, takes a single column from a dataset and then adds 50 anomalies to the dataset that is quite bigger than the maximum values of the dataset. ...
0
votes
1answer
15 views

How to model anomaly data using Gaussian distribution assuming variables are dependent? (In Python)

I have some data which contains anomalies as well. I want to model data using Gaussian distribution assuming variables are dependent in Python. How can I model this? Should I use the PDF formula as ...
0
votes
0answers
12 views

How to set anomaly threshold depending of predictive model accuracy

Say I have a variable with a standard deviation STD I have a predictive model to predict variable. The model accuracy is 80% An anomaly is raised if difference (predicted_value - actual value) > ...

1
2 3 4 5