Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0 votes
1 answer
6 views

RCA on top of existing anomalies

When we work with timeseries data containing multiple features eg. a sensor data. we can detect anomalies using cluster, supervised and semi supervised based approaches (Eg. Isolation, Autoencoder Etc)...
  • 111
0 votes
1 answer
8 views

Unsupervised anomaly detection - dataset with multiple users

I'm confused on how to go about an issue. I'm trying to implement an unsupervised model, using a dataset that is essentially a log file. This dataset contains a variety of features, but most ...
  • 1
1 vote
0 answers
30 views

Threshold for auto encoder anomaly detection

I have fitted an auto encoder on 25-dimensional time series data hoping to be able to detect anomalies. training set is 100k observations, testset for threshold setting is 10k observations. all ...
  • 21
0 votes
0 answers
19 views

Cluster-Based Anomaly Detection (and PCA)

I have a dataset of user operations (250 types of operations) in a trading platform and my task is to extract features, flagging rules, other insight for fraud prevention/anomaly detection, or some ...
1 vote
1 answer
28 views

Best way to detect newly incoming anomalies in two timeseries?

I have two devices that both send data (let's say temperature). I need to be able to detect if one of the devices reports an unusual/anomalous reading. The case if both of the devices report a "...
2 votes
1 answer
38 views

Early anomaly detection / Failure prediction on time series

My problem here is that I want to predict failures in advance with respect to their occurrence. I have sensors mounted on my machine and with a certain frequency, they send data to my database. ...
  • 45
1 vote
1 answer
18 views

Underfitting and perfomance metrics in unsupervised methods

My question is simple and yet quite hard to find an answer to. In an unsupervised method, for example, when you have to reconstruct an input, how can you tell if your loss is good enough? Generally, ...
-2 votes
1 answer
32 views

How to increase , precision-recall value in your Deep learning model

I am getting good accuracy metrics around 80 with precision =66, recall =37, F1 =47. How can I improve precision, and recall metrics in anomaly detection scenarios.. any suggestions?
  • 171
1 vote
1 answer
30 views

Determining threshold for KMeans anomaly detection

I'm trying to use KMeans for anomaly detection, and I know that a threshold is needed to determine the anomalies. I've seen many articles talking about how to choose K, but none talks about how to ...
  • 131
0 votes
0 answers
17 views

Determine unusual occurrence of words in classes

I am working on a project where I have 20+ classes/groups. Each of these groups perform certain text searches. I am looking for specific keywords example 'code' which is an anomaly. The challenge is ...
0 votes
0 answers
24 views

How to detect Novelty from different ranges of target variable?

I've a dataset of multiple categorical columns along with a target column that is continuous. Assume combination of categorical columns has a different range of values of target. Ex Col1 - col2 - col3 ...
0 votes
0 answers
19 views

Video anomaly detection/ Evaluation AUC

I have trained an unsupervised anomaly detector for surveillance videos. After inference, I rescale the scores between max/min from the resulting scores array. scores = (scores - min(scores))/max(...
  • 1
0 votes
1 answer
27 views

Sound Anomaly Detection

What is the recommended directory structure for sound anomaly detection using Keras CNN (Unsupervised) ? After converting the sound files into spectrograms. Code examples will be highly appreciated.
  • 1
0 votes
1 answer
26 views

detecting abnormality in a specific feature with respect to others (unsupervised?)

I have a large dataset with a feature y which is dependent in part on features x1 and x2. All features are noisy, and y is also dependent on other parameters not captured in the dataset. I would like ...
0 votes
0 answers
12 views

What is the use of validation dataset when doing regression-based outlier detection?

I have a dataset where data are velocity data splitted as: 60%(train - non-anomalous) 20%(validation - 50% of it anomalous) 20%(test - 50% of it anomalous). From my understanding, when doing outlier ...
  • 121
4 votes
1 answer
442 views

How to detect anomalies in web log data

I have the following challenge: We have the web logs of a platform where people can download publications and we need to detect anomalies. From time to time and only by chance we observe spikes in ...
  • 143
0 votes
1 answer
18 views

Autoencoder Layers

I am using AutoEncoder to detect anomalies and my dataset is a numerical dataset that has 10 columns (including the target label), I don't know what numbers I should choose for the first argument in ...
0 votes
0 answers
7 views

Can/should a logical multi-class classification anomaly detection system be described as "unsupervised machine learning"?

I would like to ensure that my use of terminology is accurate. My question is: what terminology should I be using in this case? The system I am building assigns classes (-1, 0, +1) to observations ...
0 votes
2 answers
111 views

Time Series - Anomaly Detection

I have time-series data with alerts (every minute) that I need to find anomalies in. I am looking for a library which can do unsupervised learning of this data and detect anomalies in the data. Which ...
0 votes
0 answers
18 views

(anomaly detection) Is it wise to include human annotated y' in anomaly detection in order to detect the true outlier y?

I am doing a fraud detection project. Right now, I have feature variables and binary dependent variable y' which is labelled using heuristics. Ideally, I need to automate this manual process and ...
0 votes
0 answers
24 views

Handling features dependent on a data field that may not be present in sample

I'm trying to build an anomaly detection model using Isolation Forest. I currently have 12 features, about half of them depends on the presence of a particular data field, say ...
  • 131
0 votes
0 answers
26 views

Why anomaly detection with IForest does not give expected results?

My goal is to develop an anomaly detection model with Isolation Forest in order to distinguish between normal and anomalous IPs by analyzing the web access logs and ...
  • 101
0 votes
0 answers
13 views

How to calculate feature contribution to regressor error in an anomaly detection problem?

Say I have a standard regressor, such as an elastic net model, used to detect anomalies in a system. I am predicting target sensor T, with feature A, B, C, which are upstream sensors of sensor T. What ...
1 vote
2 answers
34 views

Validate Unsupervised Binary Classification

I’m working on a fully unsupervised anomaly detection problem. Since it’s completely unsupervised, I’m having hard times in defining some metrics to kind of validate the results (I run several ...
0 votes
0 answers
14 views

Find the most impactfuls parameters multivariate output unsupervised ML

I am currently on a proect where my df has more than 600 parameters of analog sensors (A parameters) and about 50 other parameters (F parameters). I want to find for each of these 50 parameters (F ...
  • 1
1 vote
0 answers
35 views

How to find inconsistency in data?

I wonder how I can find any inconsistencies in data. To show what I mean let's consider the following example: Imagine I have a problem of finding out why the speed (S) of the vehicle is low. I have ...
  • 121
1 vote
1 answer
38 views

How can I find anomalies in features based on difference between true and predicted targets?

Generally the problem is the following : there is target (efficiency of mobile stations). The goal is to find stations which underperform and highlight the reasons of that. Other than this it is ...
  • 121
0 votes
0 answers
19 views

Detecting anomalies/web performance on Google Analytics

I would like to detect any anomalies on my web traffic based on the data being collected by Google Analytics (GA). Simply, I have form events data on GA and they normally have the same trend as ...
0 votes
2 answers
57 views

Anomaly prediction/forecasting in timeseries?

What options exist in order to forecast when next observation will be an outlier in a time series? Initially, I thought to train a simple forecasting model, which turned out to decently predict the ...
0 votes
0 answers
27 views

Understanding Isolation Forest predictions

I'm running sklearn's IsolationForest on a dataset containing 2 classes of data, one that I know is the anomaly (~1.5% of the entire dataset), the other is the normal dataset. I'm using this (shuffled)...
  • 131
0 votes
0 answers
10 views

Trying to bootstrap code from another script i build using one-hot-encodeing, this time i don't need to encode

I have a code bit that i'm trying to duplicate except for my matches being encoded I just have a binary 0 or 1 for my data in the field that is to be indexed. If i substitute 1 or 0 for the "...
0 votes
0 answers
22 views

Dealing with conditional variables in anomaly detection

I am working on an anomaly detection model and I have a conditional variable, i.e., it is zero or it has an amount like below histogram. Suppose the variable shows the time when a machine is not ...
  • 11
0 votes
0 answers
18 views

What are the best libraries/models for time series anomaly detection?

I am working on time series anomaly detection and would like to find the latest and most efficient libraries or models for that.
0 votes
0 answers
42 views

Anomaly Detection in a time series data using statistical methods

0: I've been working on time series data recently and need to detect the anomalies observed using HoltWinters(Triple exponential smoothing) model. Base: I implemented Triple exponential smoothing, ...
0 votes
1 answer
229 views

How to fit a model on validation_data?

can you help me understand this better? I need to detect anomalies so I am trying to fit an lstm model using validation_data but the losses does not converge. Do they really need to converge? Does the ...
0 votes
1 answer
87 views

Cross-validation for anomaly detection on time series data

I want to perform k-fold cross-validation for the setting where I have a training dataset consisting of a sequential time series that is fully benign and a test dataset (also a sequential time series)...
1 vote
1 answer
86 views

is it good to have 100% accuracy on validation?

i'm still new in machine learning. currently i'm creating an anomaly detection for flight data. it is a multivariate time series data that include timestamp, latitude, longitude, velocity and altitude ...
1 vote
1 answer
116 views

How to compute threshold?

I would like to detect anomalies for univariate time series data. Most examples on internet show that, after you predict the model, you calculate a threshold for the training data and a MAE test loss ...
0 votes
1 answer
54 views

An autoencoder setup for anomaly detection

I am doing anomaly detection using machine learning. i have tried different models like isolation forest, SVM and KNN. The maximum accuracy that I can get from each of them is $80\%$ accordind to my ...
  • 11
0 votes
0 answers
21 views

Is it impossible to predict defects with data that are not labeled?

There is manufacturing data with 10 process variables. Normal and bad labeling are not done. It's tabular fdata. Do you have a paper that only uses data that are not labeled to predict defects or to ...
  • 11
0 votes
0 answers
32 views

How to detect anomalies?

I have timeseries data with one value per day for a year. (there is one column with temperature data). I am using autoencoders to train a reconstruction model with mse loss. Firstly, I normalized the ...
0 votes
0 answers
19 views

Which machine learning technique can be used for predictive log analysis

I have log data with 100k records. And These parameters. It looks like this. message types can be helpful for anomaly type detection. Out of total 15 message 5 ...
0 votes
1 answer
15 views

How to find an anomalous matrix among many?

Let's say we have a bunch of matrices that we know are non-anomalous. We now receive a new matrix and want to know if it belongs into the group or is way off. Is there a way to do that? I'm thinking ...
0 votes
0 answers
118 views

Unsupervised learning for anomaly detection based on memory and cpu usage

Recently got into Data Science, I've been working on a data science project, I have built a system that collects real-time logs on virtual machines in the cloud, the logs include memory consumption ...
0 votes
1 answer
151 views

How can I generate anomalies in a dataset?

I'm building a tensorflow model to detect anomalies in an electricity smart meter data and I'm using UK-DALE Dataset. How can I introduce anomalies in the data so I can test the model?
0 votes
0 answers
128 views

Confidence/prediction interval vs lower and upper bounds in ARIMA_PLUS big query

How is CI intervals in ML.forecast and lower-upper bounds in ML.Detect_Anomalies different for Arima_plus model in BigqueryML ?
1 vote
1 answer
160 views

Training data for anomaly detection using LSTM Autoencoder

I am building an time-series anomaly detection engine using LSTM autoencoder. I read this article where the author suggests to train the model on clean data only in response to a comment. However, in ...
0 votes
1 answer
108 views

regarding computing the centroid of high dimensional data

In scikit-learn, or other python libraries, are there any existing implementations to compute centroid for high dimensional data sets?
1 vote
0 answers
22 views

Multiple Timeseries Anomaly Detection - identifying which feature is anomalous?

I have a multivariate time series where I have features such as: temperature set point energy used relative humidity, etc. Currently, I'm creating univariate anomaly detection models in Python using ...
1 vote
1 answer
256 views

how to select threshold for unsupervised anomaly detection

I am working on an anomaly detection use case. I studied one technique of selecting the threshold that marks 5% of validation data as anomalies. how it works in anomaly detection cases. and there is ...
  • 171

1
2 3 4 5
7