Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0 votes
1 answer
93 views

What is the best practice to normalize/standardize imbalanced data for outlier detection or binary classification task?

I'm researching Anomaly/outlier/fraud detection, and I'm looking for the best practice to pre-process the synthetic data for imbalanced data. I have checked all methodology for normalizing/...
0 votes
0 answers
10 views

What are the best libraries/models for time series anomaly detection?

I am working on time series anomaly detection and would like to find the latest and most efficient libraries or models for that.
2 votes
0 answers
297 views

Comparison between approaches for timeseries anomaly detection

After various days of research, I could take a global picture of the existing methods to perform anomaly detection on time series, namely: Forecasting with Deep Learning. Eg. RADM or LSTM model ...
0 votes
1 answer
110 views

How to gps data anomaly detection in python

I have gps format dataset lat, lon. I want to detection anomaly using python. I tested knn, smv, cof, iforest using pycaret. But i did not. These colors anomlay because the angle change is too much ...
5 votes
1 answer
3k views

Isolation Forest Feature Importance

As of scikit-learn version 0.19.1, there is no implementation for calculating feature importance in an Isolation Forest. I'm also having trouble finding any online resources proposing ways to get at ...
0 votes
0 answers
16 views

Anomaly Detection in a time series data using statistical methods

0: I've been working on time series data recently and need to detect the anomalies observed using HoltWinters(Triple exponential smoothing) model. Base: I implemented Triple exponential smoothing, ...
0 votes
1 answer
315 views

Algorithms for Anomaly Detection of Event Sequence Data [Python/R]

I am building an anomaly detection system of event sequence data (transactions). For each timestep, a transaction can be in any of 76 different stages. My dataset is therefore a 3D array of size(m,t,N)...
0 votes
1 answer
43 views

Word2vec to encode medical procedures when using isolation forests

I am planning to use Isolation Forests in R (solitude package) to identify outlier medical claims in my data. Each row of my data represents the group of drugs that each provider has administered in ...
0 votes
2 answers
68 views

Anomaly Detection

I have a problem where I want to identify Vendors with unusual high amount invoices. What would be the best way to identify such invoices? I am trying to use Isolation Forest but having trouble in ...
4 votes
1 answer
71 views

anomaly detection in vehicle sensor data

I am currently diving deeper into understanding more about anomaly detection in regards to vehicle's data generated by sensors. It seems like there is no proper book or article that goes deeper into ...
2 votes
1 answer
85 views

Accept any suggestion to create training data from correlation matrix to find odd one out to identify difference in variation

I have N time varying feature vectors obtained by recording different parameters over time.This results in N*N similarity matrix which contains one to one correlations value for each feature. We need ...
0 votes
1 answer
41 views

which statistical parameters are more useful to detect anomalies and outlier? mean max min var?

This time series contains some time frame which each of them are 8K (frequencies)*151 (time samples) in 0.5 sec [overall 1.2288 millions samples per half a second) I need to find anomalous based on ...
0 votes
1 answer
30 views

How to fit a model on validation_data?

can you help me understand this better? I need to detect anomalies so I am trying to fit an lstm model using validation_data but the losses does not converge. Do they really need to converge? Does the ...
0 votes
1 answer
22 views

Cross-validation for anomaly detection on time series data

I want to perform k-fold cross-validation for the setting where I have a training dataset consisting of a sequential time series that is fully benign and a test dataset (also a sequential time series)...
1 vote
1 answer
49 views

Labels as features in anomaly detection

I have a dataset born to solve a classification problem. Due to the imbalances of the Y, i choose to move to an anomaly detection task. Should I use the Y i have inside the anomaly detection model as ...
1 vote
1 answer
298 views

Isolation Forest Score Function Theory

I am currently reading this paper on isolation forests. In the section about the score function, they mention the following. For context, $h(x)$ is definded as the path length of a data point ...
1 vote
1 answer
48 views

An Unsupervised learning method suitable for large categorical data sets

I want to detect anomalies in the bank data set in an unsupervised learning method. However, in the bank data set, all columns except time and amount were categorical data, and about half of them had ...
1 vote
1 answer
83 views

Anomaly detection and replacing it with past values in time series

I am trying to use anomaly detection to find the anomalies in my time series, and if I find it, I will replace it with my past values. I'm trying to do this because I want to create an upper and lower ...
0 votes
1 answer
50 views

Incremental learning on Autoencoder for anomaly detection

I want to incrementally train my pre-trained autoencoder model on data being received every minute. Based on this thread, successive calls to model.fit will incrementally train the model. However, the ...
0 votes
1 answer
107 views

Decision trees for anomaly detection

Problem From what I understand, a common method in anomaly detection consists in building a predictive model trained on non-anomalous training data, and perform anomaly detection using the error of ...
1 vote
1 answer
34 views

is it good to have 100% accuracy on validation?

i'm still new in machine learning. currently i'm creating an anomaly detection for flight data. it is a multivariate time series data that include timestamp, latitude, longitude, velocity and altitude ...
1 vote
1 answer
45 views

How to compute threshold?

I would like to detect anomalies for univariate time series data. Most examples on internet show that, after you predict the model, you calculate a threshold for the training data and a MAE test loss ...
2 votes
1 answer
319 views

Anomaly detection - relation between thresholds and anomalies

I'm developing an anomaly detection program in Python. Main idea is to create a new LSTM model every day, training it with the previous 7 days and predict the next day. Then, using thresholds, find ...
0 votes
0 answers
24 views

An autoencoder setup for anomaly detection

I am doing anomaly detection using machine learning. i have tried different models like isolation forest, SVM and KNN. The maximum accuracy that I can get from each of them is $80\%$ accordind to my ...
0 votes
0 answers
20 views

Is it impossible to predict defects with data that are not labeled?

There is manufacturing data with 10 process variables. Normal and bad labeling are not done. It's tabular fdata. Do you have a paper that only uses data that are not labeled to predict defects or to ...
0 votes
1 answer
79 views

is there a way to check if i got a "good price" on something?

I'm looking at some data. Actually, the Boston Housing dataset is probably a good proxy for it: https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html I'm wondering if there's a way to ...
5 votes
2 answers
81 views

Detecting abundance of a certain periodic pattern in a time series?

I am really stumped at the moment about how to solve a particular problem. I have many time series like this: This represents the number of hours a person spends on a website each day throughout the ...
1 vote
2 answers
66 views

unsupervised anomaly detection for univariate fast frequency time series data?

I have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer This time series ...
0 votes
0 answers
27 views

How to detect anomalies?

I have timeseries data with one value per day for a year. (there is one column with temperature data). I am using autoencoders to train a reconstruction model with mse loss. Firstly, I normalized the ...
1 vote
1 answer
87 views

how to set threshold for anomaly detection

I read one research paper and they said that they are using a threshold for anomaly detection. The threshold is determined to make some proportion of data of the validation dataset labeled as ...
0 votes
0 answers
11 views

Which machine learning technique can be used for predictive log analysis

I have log data with 100k records. And These parameters. It looks like this. message types can be helpful for anomaly type detection. Out of total 15 message 5 ...
-1 votes
2 answers
168 views

huge doubt on anomaly detection

from the naked eye itself, we can tell in the region 5161 the network usage is high so that is the anomaly in my case, then why do we want to apply k-means and other machine learning algorithms to ...
0 votes
1 answer
327 views

Sampling labeled data for anomaly detection

I'm currently working on a project that requires the use of unsupervised anomaly detection, but I'm unable to find a relevant data set, so I'm considering the following option: Assuming I have a data ...
5 votes
2 answers
285 views

Anomaly detection using clustering of highly correlated Categorical data

My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else it's Anomaly. Likewise, there are thousands ...
0 votes
1 answer
13 views

How to find an anomalous matrix among many?

Let's say we have a bunch of matrices that we know are non-anomalous. We now receive a new matrix and want to know if it belongs into the group or is way off. Is there a way to do that? I'm thinking ...
2 votes
1 answer
54 views

What's the best way to validate a rare event detection model during training?

When training a deep model for rare event detection (e.g. sound of an alarm in a home device audio stream), is it best to use a balanced validation set (50% alarm, 50% normal) to determine early ...
1 vote
1 answer
212 views

Algorithm suggestion for anomaly detection in multivariate time series data

I have time series data containing user actions at certain time intervals eg ...
0 votes
1 answer
95 views

how to set threshold value by looking at loss distribution in anomaly detection task

I am following this tutorial https://towardsdatascience.com/lstm-autoencoder-for-anomaly-detection-e1f4f2ee7ccf to use LSTM autoencoder to detect anomalies in my unsupervised dataset. they plotted ...
0 votes
2 answers
777 views

Percentile as a threshold for Anomaly Detection?

I'm following this article about Unsupervised Anomaly Detection Algorithms. In this article, a threshold value is calculated using the scipy score percentile method to determine whether the point is ...
0 votes
1 answer
135 views

Alternative methods for novelty detection and correlations

Hey mates I have the following project: Imagine having two datasets A and B. Each dataset consits of 101 time series with the ...
3 votes
1 answer
1k views

IsolationForest Decision Function vs. Anomaly Prediction Question

I'm currently working on an unsupervised anomaly detection project, and for it I'm using IsolationForest through scikit-learn. My question is, why/how is it possible for the model to predict something ...
0 votes
0 answers
20 views

Unsupervised learning for anomaly detection based on memory and cpu usage

Recently got into Data Science, I've been working on a data science project, I have built a system that collects real-time logs on virtual machines in the cloud, the logs include memory consumption ...
0 votes
1 answer
27 views

How can I generate anomalies in a dataset?

I'm building a tensorflow model to detect anomalies in an electricity smart meter data and I'm using UK-DALE Dataset. How can I introduce anomalies in the data so I can test the model?
0 votes
0 answers
19 views

Confidence/prediction interval vs lower and upper bounds in ARIMA_PLUS big query

How is CI intervals in ML.forecast and lower-upper bounds in ML.Detect_Anomalies different for Arima_plus model in BigqueryML ?
1 vote
1 answer
246 views

K-Prototype for anomaly detection

I have logs of the form (e.g. from a gym login).. the representational case is so: UserName, Login time, timeSpend_on_weights, time_spent_on_elliptical ...
2 votes
1 answer
163 views

Dynamic clustering

I am performing anomaly detection on different datasets and thought to first cluster the dataset and submit each of the clusters to different AD models. I am using HDBSCAN, and in my test dataset I ...
0 votes
1 answer
31 views

Training data for anomaly detection using LSTM Autoencoder

I am building an time-series anomaly detection engine using LSTM autoencoder. I read this article where the author suggests to train the model on clean data only in response to a comment. However, in ...
1 vote
1 answer
315 views

Forecasting vs non-forecasting predition for time series anomaly detection

I have got the objective of implementing a uni/multivariate online anomaly detection system. After multiple days of research, I could collect many ways to achieve this (Eg. moving average solutions ...
2 votes
2 answers
1k views

How can I find anomalies in each row of data?

I have some reported data I want to spot anomalies on. The columns are a facility name then monthly reports of that given facility. ...
0 votes
1 answer
22 views

regarding computing the centroid of high dimensional data

In scikit-learn, or other python libraries, are there any existing implementations to compute centroid for high dimensional data sets?

1
2 3 4 5
7