Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Statistical method to validate predicted outliers

I was trying to make an clustering based unsupervised anomaly detection on a large high dimensional dataset. Roughly saying the points not lied inside all the clusters are defined as anomalies or ...
-1
votes
1answer
30 views

Anomaly Detection and Removal/Interpolate

I am performing a machine learning regression task on time series data. I have a data frame filled with the close prices of various assets and economic data. I am looking to perform outlier detection ...
1
vote
1answer
29 views

In general, what are precision, recall, F1 that are reported in papers?

I used classification_report in sklearn library And, the picture below shows evaluation on my model (anomaly detector) In general, what are precision, recall, F1 ...
1
vote
2answers
31 views

unsupervised anomaly detection for univariate fast frequency time series data?

I have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer This time series ...
0
votes
0answers
13 views

identifying time series with threshold breach potential

(moved from stackoverflow.com) Hi all, I'm trying to solve a following problem. I have a set of various devices feeding their readings into a system where they are stored as time series: timestamp, ...
0
votes
0answers
19 views

How is convex hull method used in outlier detection?

I think the slides are bit unclear on what they want to say. Can someone elaborate this with example.
0
votes
1answer
27 views

Principal Components of PCA

I have a dataset combined with normals samples and fault samples. We use PCA on normal samples and faults samples separately. I observe that the last PCs help us to have better separation than for ...
0
votes
1answer
28 views

Evaluation metric for imbalanced data

Hi I'm a CS graduate student I have a question for AI or data experts. I'm writing a paper My dataset is time-series sensor data and anomaly (positive class) ratio is between 5% and 6% you can see the ...
1
vote
0answers
23 views

Using STL(Seasonal-Trend decomposition using LOESS) for Anomaly detection

I am using STL to decompose my time series data in Season, trend and residual and then by applying this(see below) on residual. I am detecting the anomaly ...
1
vote
1answer
29 views

How to test unsupervised learning methods for anomaly detection?

How to test unsupervised learning methods for anomaly detection? I am looking for a test strategy to evaluate my result of my anomaly detection technique? what is your offer more than evaluate with ...
1
vote
1answer
26 views

Evaluation metric for time-series anomaly detection

I have a question for AI or data experts. I'm writing a paper My dataset are time-series sensor data and anomaly ratio is between 5% and 6% 1. For time-series anomaly detection evaluation, which one ...
1
vote
0answers
17 views

What is the best feature extraction technique for text domain novelty / anomaly detection?

I am working with a text classification system. Here, my data-set has around 30 intents. But the problem is I have no system developed to handle inputs that don't go under any of the intents. So, in ...
0
votes
0answers
15 views

Continuous Machine Learning with Log Streams

I am doing research in Continuous Machine Learning/ Life Long Learning. Two of the use cases I came across were Predicting Failures and Anomaly Detection using log stream data. However, already there ...
0
votes
1answer
24 views

Distance between any two points after DBSCAN

DBSCAN is a clustering model which is robust to detect the outliers also. A parameter $\epsilon$ i.e. radius is an input of the algorithm, a point is said to be outlier if it's circle with radius $\...
0
votes
0answers
16 views

Anomaly Detection: LSTM Autoencoder Zero Reconstruction Loss on Anomalies

I am using an LSTM Autoencoder model for time series anomaly detection. None of the anomalies get flagged because the reconstruction loss comes out to be zero for all data points on the clearly ...
1
vote
0answers
26 views

One-class SVM formula

Recently I have been studying one-class SVM and am a little bit confused about the offset $\rho$. The common optimization problem is to find a function $f(x)= w^\top x-\rho$ by solving $$\begin{array}...
0
votes
0answers
16 views

combine multiple datasets for generating a model

I am working on anomaly detection and have a dataset with multiple graphs. For instance, one graph has 100K edges and it belongs to video game activities (API calls). Another graph belongs to attack ...
0
votes
0answers
21 views

Data preprocessing - Time Series data resets its values - Detection & Correction

1. Summarize the problem I currently trying to work with time series data from sensors which has some problems regarding resetting it values. For example some cumulative values gets reset and don't ...
0
votes
1answer
22 views

Real-Time Outlier/Anomaly Detection?

My data is the usage/playing statistics for players of a specific game. One data point for a user is aggregated statistics for one week. The goal is to be able to detect when the account of the player ...
0
votes
0answers
14 views

Find minimal group of 'spanning items'

I have a group of similarity hashes which I use to detect similar items using hamming_distance < 10. Each time a new item is being detected by one of the item ...
0
votes
1answer
46 views

What is the best practice to normalize/standardize imbalanced data for outlier detection or binary classification task?

I'm researching Anomaly/outlier/fraud detection, and I'm looking for the best practice to pre-process the synthetic data for imbalanced data. I have checked all methodology for normalizing/...
0
votes
0answers
32 views

ValueError: Input 0 of layer conv_lst_m2d_60 is incompatible with the layer: expected ndim=5, found ndim=4. Full shape received: (None, 7, 7, 512)

I am building an anomaly detection model using keras upon videos. There are total 179 frames. The original dimension of each frame is given below: ...
1
vote
1answer
23 views

Train classifier to detect crest of wave

In the picture, there are data and a target in the time series. The data is padded until it reaches the max length. The target is marked by humans. It pins down the starting point and stopping point ...
1
vote
0answers
24 views

An Unsupervised learning method suitable for large categorical data sets

I want to detect anomalies in the bank data set in an unsupervised learning method. However, in the bank data set, all columns except time and amount were categorical data, and about half of them had ...
1
vote
1answer
12 views

Practical problems in anomaly detection where the number of normal data is extremely high compared to abnormal data

If the ratio of abnormal data is about 1 to 10,000 normal data, even if the true negative rate is 99%, there will be 100 false positive data, and the precision( TP/(TP+FP) ) will be low. If this kind ...
2
votes
1answer
68 views

The affect of bootstrap on Isolation Forest

I've been using isolation forest for anomaly detection, and reviewing its parameters at scikit-learn (link). Looking at "bootstrap", I'm not quite clear what using bootstrap would cause. For ...
0
votes
0answers
29 views

Input pipeline with an autoencoder and tf.data

I am using an autoencoder to detect anomalies in dataset of network traffic. The dataset is a csv file, and is loaded and preprocessed with pandas (encoded categorical features with pandas.get_dummies(...
0
votes
0answers
12 views

Anomaly Detection in Highly Variable Time-Series Data

I am trying to detect anomalies through a column called count. The data is a time-series data and it is present for every 5 minutes for each day. The dataframe looks like this: ...
1
vote
0answers
49 views

Anomaly detection for high dimensional categorical data

I have a dataset with around 200+ categorical variables $X_i$ and the sizes of their domain $|X_i|$ range from 2 to 8k. So, if I one-hot encode the combination of these variables, the vector space (...
1
vote
0answers
31 views

Interpreting one-class SVM

I am new to SVM (one-class) and was practically investigating it. Got some weird result that I can not explain. Let me demonstrate by some small reproducible code and visualization: ...
0
votes
0answers
76 views

How can interparet shap.summary_plot and its gray color concerning outliers/anomaly?

I inspired by this notebook, and I'm experimenting IsolationForest algorithm using scikit-learn==0.22.2.post1 for anomaly ...
1
vote
1answer
69 views

Similarity between 2 statistical distributions

Is there any index that measures similarity between 2 gaussian distributions of 1-D data (may have slightly different number of points) considering their mean shift, variance shift, difference in ...
1
vote
0answers
30 views

How to calculate MAE and threshold in a multivariate time series

I'm trying to understand how to calculate the MAE in my time series and then the thresholds to understand which of my data in the test set are anomalies. I'm following this tutorial, which is based on ...
1
vote
2answers
111 views

Autenocoder and anomaly detection task

I'm trying to create an autoencoder for the anomaly detection task, but I'm noticing that even if it performs very well on the training set, it starts to stop recreating half of the test set. I tried ...
0
votes
1answer
58 views

Understanding time series anomaly detection using Autoencoder

I'm studying how to detect anomalies in the time series using an Autoeconder. In particular, I'm following the guide posted in the Keras website, but I don't understand why they are creating and how ...
0
votes
1answer
45 views

Anomaly Detection

I have a problem where I want to identify Vendors with unusual high amount invoices. What would be the best way to identify such invoices? I am trying to use Isolation Forest but having trouble in ...
1
vote
0answers
14 views

Finding optimal time series using convolution [closed]

we logged sensor data while milling a workpiece. At several points, the workpiece was damaged and this induced a certain sensor data time series. Due to noise and since its a real world measurement, ...
1
vote
1answer
60 views

Anomaly detection on sparse categorical data

I have a big dataset with a column "clientid" and a categorical column "choice". I want to find out what are the clients that have strange combinations of choices (less frequent ...
0
votes
0answers
33 views

Convolutional AE always overfitting time series - what’s wrong?

I've build a CAE for anomaly detection in time series, but it is always overfitting. I've tried data augmentation, short/long inputvector, dropout rates... I don't know what I'm doing wrong, may be ...
1
vote
0answers
16 views

How to preprocess data for multiple rows instance? [closed]

I have a machine learning problem where I must detect anomalies in (let say) taxes declaration. So I have multiple rows for each enterprise, each row describing the tax declaration a the time (date,...
1
vote
1answer
51 views

Geolocation Based Anomaly Detection in IPs Using Isolation Forest

I'm trying to detect anomalies based on geolocation from IP addresses on a server access log file. I have created two features country and geo_velocity, using the IP address and the timestamp of each ...
0
votes
1answer
25 views

Performing anomalie detection on a battery volatge using LSTM-RNN

I am trying to detect anomalies in a battery output voltage for one month. I have the next data frame, as it is shown the data is collected each minute for each day so I have almost 1420 sample per ...
0
votes
1answer
18 views

(Labeled, if possible) time-series datasets for anomaly detection [closed]

I would like to create a big list of available time-series datasets for anomaly detection. I'm especially interested in the following: The time-series data should be segmented into cycles Ideally, ...
0
votes
1answer
42 views

How to find anomalies in (almost) constant stream of data?

I have a process that (simply put), starts every 5 minutes, collects data, and put that data into the database. More detailed explanation would be that process starts, collects data (which takes some ...
3
votes
1answer
43 views

How to determine the abnormality of a specific variable by taking into account all the other variables in the data?

I have an issue of machine learning/anomaly detection. Indeed, I have a variable Y and several other variables X. The purpose is to quantify the degree of abnormality of the data on Y but I have to ...
0
votes
0answers
17 views

How to do parameter estimation on relatively few samples?

Context: I have a time series on which I've trained an autoencoder for anomaly detection (AD) purposes. My time-series is 6 years worth of daily data, but for AD purposes, I am looking at one week ...
0
votes
0answers
71 views

outliers in time series

I have a data set like the following where the rows are dates and the columns are values recorded by different sensors on those dates. Before working with the data for the purpose of predicting it, I ...
0
votes
0answers
61 views

How can realize the evaluation/validation of unsupervised models through unlabeled data?

I'm researching anomaly detection, which is nothing else than outliers detection on a set of time-series web servers access log data or network traffic. Recently I re-faced to following fundamental ...
1
vote
1answer
38 views

Labels as features in anomaly detection

I have a dataset born to solve a classification problem. Due to the imbalances of the Y, i choose to move to an anomaly detection task. Should I use the Y i have inside the anomaly detection model as ...

1
2 3 4 5 6