Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

What is the name of this technique involving tracking cumulative errors with a forgiveness parameter?

I'm looking for the name of a technique I've seen used before. Most common in time-series based anomaly detection. It involves keeping a running total of consecutive "error" amounts, ...
0
votes
0answers
10 views

Clustering 3D image voxels based on their location and value

My goal is to detect whether a MRI image contains an anomaly and the location of the anomaly. In my dataset I have MRI brain images which contain values of electrical conductivity of brain tissues. ...
0
votes
0answers
30 views

How to evaluate unsupervised Anomaly Detection using k-means

I'm trying out different anomaly detection models and would love to hear opinion on my idea from somebody experienced. My goal is to perform anomaly detection with different models and to give each ...
1
vote
2answers
42 views

How do I determine the top "reason" for anomaly when using Isolation Forests

I am using Isolation Forests for Anomaly Detection. Say, my set has 10 variables, var1, var2, ..., var10, and I found an anomaly. Can I rank the 10 variables var1, var2, ..., var10 in such a way I can ...
0
votes
1answer
34 views

which statistical parameters are more useful to detect anomalies and outlier? mean max min var?

This time series contains some time frame which each of them are 8K (frequencies)*151 (time samples) in 0.5 sec [overall 1.2288 millions samples per half a second) I need to find anomalous based on ...
0
votes
1answer
71 views

Anomaly (Outlier) Detection with Isolation Forest too sensitive even with low contamination

I'm trying to use the sklearn implementation of the Isolation Forest algorithm to detect anomalies in my time series data. However, even with a very low contamination parameter (0.0001), it is ...
0
votes
0answers
12 views

What is the range of Anomaly_Score column when using PyCaret anomaly detection library?

When we use the PyCaret library for anomaly detection we import it using from pycaret.anomaly import * and we create a model using the ...
0
votes
0answers
10 views

Weight for Samples on SVM

there is a option sample_weight in fit(X[, y, sample_weight]) function (OneClassSVM, sklearn library). If I use the option ...
1
vote
1answer
36 views

How can I weight each point in one-class SVM?

I want to give weights to some data points Specifically, these are points related to anomalies (I'm implementing one-class SVM for anomaly detection) Exactly, I want to consider some data points that ...
0
votes
0answers
16 views

Statistical method to validate predicted outliers

I was trying to make a clustering-based unsupervised anomaly detection on a large high-dimensional dataset. Roughly saying the points not lied inside all the clusters are defined as anomalies or ...
-1
votes
1answer
47 views

Anomaly Detection and Removal/Interpolate [closed]

I am performing a machine learning regression task on time series data. I have a data frame filled with the close prices of various assets and economic data. I am looking to perform outlier detection ...
1
vote
1answer
37 views

In general, what are precision, recall, F1 that are reported in papers?

I used classification_report in sklearn library And, the picture below shows evaluation on my model (anomaly detector) In general, what are precision, recall, F1 ...
1
vote
2answers
58 views

unsupervised anomaly detection for univariate fast frequency time series data?

I have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer This time series ...
0
votes
0answers
14 views

identifying time series with threshold breach potential

(moved from stackoverflow.com) Hi all, I'm trying to solve a following problem. I have a set of various devices feeding their readings into a system where they are stored as time series: timestamp, ...
0
votes
0answers
19 views

How is convex hull method used in outlier detection?

I think the slides are bit unclear on what they want to say. Can someone elaborate this with example.
0
votes
1answer
36 views

Evaluation metric for imbalanced data

Hi I'm a CS graduate student I have a question for AI or data experts. I'm writing a paper My dataset is time-series sensor data and anomaly (positive class) ratio is between 5% and 6% you can see the ...
1
vote
0answers
55 views

Using STL(Seasonal-Trend decomposition using LOESS) for Anomaly detection

I am using STL to decompose my time series data in Season, trend and residual and then by applying this(see below) on residual. I am detecting the anomaly ...
1
vote
1answer
36 views

How to test unsupervised learning methods for anomaly detection?

How to test unsupervised learning methods for anomaly detection? I am looking for a test strategy to evaluate my result of my anomaly detection technique? what is your offer more than evaluate with ...
1
vote
1answer
30 views

Evaluation metric for time-series anomaly detection

I have a question for AI or data experts. I'm writing a paper My dataset are time-series sensor data and anomaly ratio is between 5% and 6% 1. For time-series anomaly detection evaluation, which one ...
1
vote
0answers
18 views

What is the best feature extraction technique for text domain novelty / anomaly detection?

I am working with a text classification system. Here, my data-set has around 30 intents. But the problem is I have no system developed to handle inputs that don't go under any of the intents. So, in ...
0
votes
0answers
15 views

Continuous Machine Learning with Log Streams

I am doing research in Continuous Machine Learning/ Life Long Learning. Two of the use cases I came across were Predicting Failures and Anomaly Detection using log stream data. However, already there ...
0
votes
1answer
26 views

Distance between any two points after DBSCAN

DBSCAN is a clustering model which is robust to detect the outliers also. A parameter $\epsilon$ i.e. radius is an input of the algorithm, a point is said to be outlier if it's circle with radius $\...
0
votes
0answers
17 views

Anomaly Detection: LSTM Autoencoder Zero Reconstruction Loss on Anomalies

I am using an LSTM Autoencoder model for time series anomaly detection. None of the anomalies get flagged because the reconstruction loss comes out to be zero for all data points on the clearly ...
1
vote
0answers
29 views

One-class SVM formula

Recently I have been studying one-class SVM and am a little bit confused about the offset $\rho$. The common optimization problem is to find a function $f(x)= w^\top x-\rho$ by solving $$\begin{array}...
0
votes
0answers
17 views

combine multiple datasets for generating a model

I am working on anomaly detection and have a dataset with multiple graphs. For instance, one graph has 100K edges and it belongs to video game activities (API calls). Another graph belongs to attack ...
0
votes
0answers
23 views

Data preprocessing - Time Series data resets its values - Detection & Correction

1. Summarize the problem I currently trying to work with time series data from sensors which has some problems regarding resetting it values. For example some cumulative values gets reset and don't ...
0
votes
1answer
25 views

Real-Time Outlier/Anomaly Detection?

My data is the usage/playing statistics for players of a specific game. One data point for a user is aggregated statistics for one week. The goal is to be able to detect when the account of the player ...
0
votes
0answers
14 views

Find minimal group of 'spanning items'

I have a group of similarity hashes which I use to detect similar items using hamming_distance < 10. Each time a new item is being detected by one of the item ...
0
votes
1answer
60 views

What is the best practice to normalize/standardize imbalanced data for outlier detection or binary classification task?

I'm researching Anomaly/outlier/fraud detection, and I'm looking for the best practice to pre-process the synthetic data for imbalanced data. I have checked all methodology for normalizing/...
0
votes
0answers
67 views

ValueError: Input 0 of layer conv_lst_m2d_60 is incompatible with the layer: expected ndim=5, found ndim=4. Full shape received: (None, 7, 7, 512)

I am building an anomaly detection model using keras upon videos. There are total 179 frames. The original dimension of each frame is given below: ...
1
vote
1answer
23 views

Train classifier to detect crest of wave

In the picture, there are data and a target in the time series. The data is padded until it reaches the max length. The target is marked by humans. It pins down the starting point and stopping point ...
1
vote
0answers
26 views

An Unsupervised learning method suitable for large categorical data sets

I want to detect anomalies in the bank data set in an unsupervised learning method. However, in the bank data set, all columns except time and amount were categorical data, and about half of them had ...
1
vote
1answer
16 views

Practical problems in anomaly detection where the number of normal data is extremely high compared to abnormal data

If the ratio of abnormal data is about 1 to 10,000 normal data, even if the true negative rate is 99%, there will be 100 false positive data, and the precision( TP/(TP+FP) ) will be low. If this kind ...
2
votes
1answer
128 views

The affect of bootstrap on Isolation Forest

I've been using isolation forest for anomaly detection, and reviewing its parameters at scikit-learn (link). Looking at "bootstrap", I'm not quite clear what using bootstrap would cause. For ...
0
votes
0answers
34 views

Input pipeline with an autoencoder and tf.data

I am using an autoencoder to detect anomalies in dataset of network traffic. The dataset is a csv file, and is loaded and preprocessed with pandas (encoded categorical features with pandas.get_dummies(...
0
votes
0answers
12 views

Anomaly Detection in Highly Variable Time-Series Data

I am trying to detect anomalies through a column called count. The data is a time-series data and it is present for every 5 minutes for each day. The dataframe looks like this: ...
1
vote
0answers
63 views

Anomaly detection for high dimensional categorical data

I have a dataset with around 200+ categorical variables $X_i$ and the sizes of their domain $|X_i|$ range from 2 to 8k. So, if I one-hot encode the combination of these variables, the vector space (...
1
vote
0answers
43 views

Interpreting one-class SVM

I am new to SVM (one-class) and was practically investigating it. Got some weird result that I can not explain. Let me demonstrate by some small reproducible code and visualization: ...
1
vote
1answer
228 views

How can interparet shap.summary_plot and its gray color concerning outliers/anomaly?

I inspired by this notebook, and I'm experimenting IsolationForest algorithm using scikit-learn==0.22.2.post1 for anomaly ...
1
vote
1answer
77 views

Similarity between 2 statistical distributions

Is there any index that measures similarity between 2 gaussian distributions of 1-D data (may have slightly different number of points) considering their mean shift, variance shift, difference in ...
1
vote
0answers
39 views

How to calculate MAE and threshold in a multivariate time series

I'm trying to understand how to calculate the MAE in my time series and then the thresholds to understand which of my data in the test set are anomalies. I'm following this tutorial, which is based on ...
1
vote
2answers
116 views

Autenocoder and anomaly detection task

I'm trying to create an autoencoder for the anomaly detection task, but I'm noticing that even if it performs very well on the training set, it starts to stop recreating half of the test set. I tried ...
0
votes
1answer
82 views

Understanding time series anomaly detection using Autoencoder

I'm studying how to detect anomalies in the time series using an Autoeconder. In particular, I'm following the guide posted in the Keras website, but I don't understand why they are creating and how ...
0
votes
1answer
54 views

Anomaly Detection

I have a problem where I want to identify Vendors with unusual high amount invoices. What would be the best way to identify such invoices? I am trying to use Isolation Forest but having trouble in ...
1
vote
0answers
15 views

Finding optimal time series using convolution [closed]

we logged sensor data while milling a workpiece. At several points, the workpiece was damaged and this induced a certain sensor data time series. Due to noise and since its a real world measurement, ...
1
vote
1answer
70 views

Anomaly detection on sparse categorical data

I have a big dataset with a column "clientid" and a categorical column "choice". I want to find out what are the clients that have strange combinations of choices (less frequent ...
0
votes
0answers
44 views

Convolutional AE always overfitting time series - what’s wrong?

I've build a CAE for anomaly detection in time series, but it is always overfitting. I've tried data augmentation, short/long inputvector, dropout rates... I don't know what I'm doing wrong, may be ...
1
vote
1answer
61 views

Geolocation Based Anomaly Detection in IPs Using Isolation Forest

I'm trying to detect anomalies based on geolocation from IP addresses on a server access log file. I have created two features country and geo_velocity, using the IP address and the timestamp of each ...
0
votes
1answer
25 views

Performing anomalie detection on a battery volatge using LSTM-RNN

I am trying to detect anomalies in a battery output voltage for one month. I have the next data frame, as it is shown the data is collected each minute for each day so I have almost 1420 sample per ...

1
2 3 4 5 6