Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

Anomaly detection with Deep learning method

For anomaly detection using an image, several method based on DNN are proposed such as CNN, AutoEncoder, GAN, etc. I found this web page: https://paperswithcode.com/sota/anomaly-detection-on-mvtec-ad ...
0 votes
0 answers
22 views

Why anomaly detection with IForest does not give expected results?

My goal is to develop an anomaly detection model with Isolation Forest in order to distinguish between normal and anomalous IPs by analyzing the web access logs and ...
  • 101
0 votes
0 answers
9 views

How to calculate feature contribution to regressor error in an anomaly detection problem?

Say I have a standard regressor, such as an elastic net model, used to detect anomalies in a system. I am predicting target sensor T, with feature A, B, C, which are upstream sensors of sensor T. What ...
1 vote
2 answers
24 views

Validate Unsupervised Binary Classification

I’m working on a fully unsupervised anomaly detection problem. Since it’s completely unsupervised, I’m having hard times in defining some metrics to kind of validate the results (I run several ...
0 votes
0 answers
13 views

Find the most impactfuls parameters multivariate output unsupervised ML

I am currently on a proect where my df has more than 600 parameters of analog sensors (A parameters) and about 50 other parameters (F parameters). I want to find for each of these 50 parameters (F ...
  • 1
1 vote
0 answers
24 views

How to find inconsistency in data?

I wonder how I can find any inconsistencies in data. To show what I mean let's consider the following example: Imagine I have a problem of finding out why the speed (S) of the vehicle is low. I have ...
  • 121
1 vote
1 answer
32 views

How can I find anomalies in features based on difference between true and predicted targets?

Generally the problem is the following : there is target (efficiency of mobile stations). The goal is to find stations which underperform and highlight the reasons of that. Other than this it is ...
  • 121
0 votes
0 answers
15 views

Detecting anomalies/web performance on Google Analytics

I would like to detect any anomalies on my web traffic based on the data being collected by Google Analytics (GA). Simply, I have form events data on GA and they normally have the same trend as ...
0 votes
2 answers
48 views

Anomaly prediction/forecasting in timeseries?

What options exist in order to forecast when next observation will be an outlier in a time series? Initially, I thought to train a simple forecasting model, which turned out to decently predict the ...
0 votes
0 answers
18 views

Understanding Isolation Forest predictions

I'm running sklearn's IsolationForest on a dataset containing 2 classes of data, one that I know is the anomaly (~1.5% of the entire dataset), the other is the normal dataset. I'm using this (shuffled)...
  • 111
0 votes
0 answers
9 views

Trying to bootstrap code from another script i build using one-hot-encodeing, this time i don't need to encode

I have a code bit that i'm trying to duplicate except for my matches being encoded I just have a binary 0 or 1 for my data in the field that is to be indexed. If i substitute 1 or 0 for the "...
0 votes
0 answers
18 views

Dealing with conditional variables in anomaly detection

I am working on an anomaly detection model and I have a conditional variable, i.e., it is zero or it has an amount like below histogram. Suppose the variable shows the time when a machine is not ...
  • 11
0 votes
0 answers
14 views

What are the best libraries/models for time series anomaly detection?

I am working on time series anomaly detection and would like to find the latest and most efficient libraries or models for that.
0 votes
0 answers
23 views

Anomaly Detection in a time series data using statistical methods

0: I've been working on time series data recently and need to detect the anomalies observed using HoltWinters(Triple exponential smoothing) model. Base: I implemented Triple exponential smoothing, ...
0 votes
1 answer
74 views

How to fit a model on validation_data?

can you help me understand this better? I need to detect anomalies so I am trying to fit an lstm model using validation_data but the losses does not converge. Do they really need to converge? Does the ...
0 votes
1 answer
43 views

Cross-validation for anomaly detection on time series data

I want to perform k-fold cross-validation for the setting where I have a training dataset consisting of a sequential time series that is fully benign and a test dataset (also a sequential time series)...
1 vote
1 answer
43 views

is it good to have 100% accuracy on validation?

i'm still new in machine learning. currently i'm creating an anomaly detection for flight data. it is a multivariate time series data that include timestamp, latitude, longitude, velocity and altitude ...
1 vote
1 answer
59 views

How to compute threshold?

I would like to detect anomalies for univariate time series data. Most examples on internet show that, after you predict the model, you calculate a threshold for the training data and a MAE test loss ...
0 votes
1 answer
43 views

An autoencoder setup for anomaly detection

I am doing anomaly detection using machine learning. i have tried different models like isolation forest, SVM and KNN. The maximum accuracy that I can get from each of them is $80\%$ accordind to my ...
  • 11
0 votes
0 answers
20 views

Is it impossible to predict defects with data that are not labeled?

There is manufacturing data with 10 process variables. Normal and bad labeling are not done. It's tabular fdata. Do you have a paper that only uses data that are not labeled to predict defects or to ...
  • 11
0 votes
0 answers
31 views

How to detect anomalies?

I have timeseries data with one value per day for a year. (there is one column with temperature data). I am using autoencoders to train a reconstruction model with mse loss. Firstly, I normalized the ...
0 votes
0 answers
15 views

Which machine learning technique can be used for predictive log analysis

I have log data with 100k records. And These parameters. It looks like this. message types can be helpful for anomaly type detection. Out of total 15 message 5 ...
0 votes
1 answer
13 views

How to find an anomalous matrix among many?

Let's say we have a bunch of matrices that we know are non-anomalous. We now receive a new matrix and want to know if it belongs into the group or is way off. Is there a way to do that? I'm thinking ...
0 votes
0 answers
51 views

Unsupervised learning for anomaly detection based on memory and cpu usage

Recently got into Data Science, I've been working on a data science project, I have built a system that collects real-time logs on virtual machines in the cloud, the logs include memory consumption ...
0 votes
1 answer
52 views

How can I generate anomalies in a dataset?

I'm building a tensorflow model to detect anomalies in an electricity smart meter data and I'm using UK-DALE Dataset. How can I introduce anomalies in the data so I can test the model?
0 votes
0 answers
56 views

Confidence/prediction interval vs lower and upper bounds in ARIMA_PLUS big query

How is CI intervals in ML.forecast and lower-upper bounds in ML.Detect_Anomalies different for Arima_plus model in BigqueryML ?
0 votes
1 answer
77 views

Training data for anomaly detection using LSTM Autoencoder

I am building an time-series anomaly detection engine using LSTM autoencoder. I read this article where the author suggests to train the model on clean data only in response to a comment. However, in ...
0 votes
1 answer
37 views

regarding computing the centroid of high dimensional data

In scikit-learn, or other python libraries, are there any existing implementations to compute centroid for high dimensional data sets?
1 vote
0 answers
18 views

Multiple Timeseries Anomaly Detection - identifying which feature is anomalous?

I have a multivariate time series where I have features such as: temperature set point energy used relative humidity, etc. Currently, I'm creating univariate anomaly detection models in Python using ...
1 vote
1 answer
126 views

how to select threshold for unsupervised anomaly detection

I am working on an anomaly detection use case. I studied one technique of selecting the threshold that marks 5% of validation data as anomalies. how it works in anomaly detection cases. and there is ...
  • 175
0 votes
0 answers
15 views

One anomaly detection model for all industries

Background - I'm creating a time-series anomaly detection (TSAD) model for the wifi throughput. My customers are 2 banks, 5 retail stores, 4 universities, 6 hospitals. Currently, I have 2 options to ...
0 votes
0 answers
44 views

Anomaly detection and root cause analysis

ARIMA is widely used for anomaly detection on time-series data e.g. stock price prediction. ARIMA assumes that future value of a variable (stock price in our case) is dependent on its previous values. ...
1 vote
2 answers
235 views

how to set threshold for anomaly detection

I read one research paper and they said that they are using a threshold for anomaly detection. The threshold is determined to make some proportion of data of the validation dataset labeled as ...
  • 175
1 vote
0 answers
18 views

Fraud detection with machine learning with partialy labeled data

I am facing difficulties handling an issue with my dataset. The purpose of my analysis is to find an electricity thief given a dataset. The first dataset: contains the list of all customers (unlabeled,...
  • 11
0 votes
0 answers
13 views

How to detect for certain malicious behavior?

I'm new to data science, and I'm trying to create a model that would help me detect certain malicious behaviors, e.g. beaconing traffic. Say I have looked at several types of beaconing traffic, and ...
  • 111
0 votes
1 answer
93 views

Incremental learning on Autoencoder for anomaly detection

I want to incrementally train my pre-trained autoencoder model on data being received every minute. Based on this thread, successive calls to model.fit will incrementally train the model. However, the ...
  • 11
0 votes
0 answers
18 views

Can I run isolation forest on existing data to find anomalies, save it for the future and use it on incoming data?

One of the major arguments I had recently is if we can save an unsupervised learning model to disk and use it later on incoming data. Isolation forest is one of the models that I use a lot for ...
0 votes
1 answer
21 views

forcasting anomaly in products

I have a question about the forecasting of anomalies. I would be very grateful if you could refer me to some papers that deal with this kind of problem or give me some hints to start with this problem....
  • 101
0 votes
0 answers
6 views

What kind of features can I obtain from IP:Port data?

I have a dataset that consist of the fields below. IP_Version,id,IP_TTL,IP_Source,TCP_Source_PORT,IP_Dest,TCP_Dest_PORT,data_size,timestamp What kind of features ...
2 votes
0 answers
155 views

Can one use PCA to reduce the dimensionality of One-Hot-Encoded data?

I read a couple times that PCA was used as a method to reduce dimensionality for one-hot-encoded data. However, there were also some comments that using PCA is not a good idea since one-hot-encoded ...
2 votes
1 answer
54 views

Various models giving 99% accuracy for KDDcup 99 dataset for Intrusion Detection, is there some sort of data leak I am missing?

Student who is quite new to all this here. I am currently working with the KDDcup 99 data for intrusion detection using various ML models (and ANN). My problem is that I am getting 99% often for ...
1 vote
1 answer
30 views

Anomaly detection for varying dictionary

I want to detect the anomaly in the processes taking up the most CPU percent. I receive the data as a time series of dictionary values like so: ...
  • 11
0 votes
0 answers
14 views

Modeling Approach for ID Persistence

I'm modeling a noisy dataset of ID's and have been tasked with creating a model to predict whether an ID exists at some arbitrary point in time. This is necessary, because an ID may not be observed at ...
0 votes
1 answer
53 views

Word2vec to encode medical procedures when using isolation forests

I am planning to use Isolation Forests in R (solitude package) to identify outlier medical claims in my data. Each row of my data represents the group of drugs that each provider has administered in ...
  • 241
0 votes
1 answer
61 views

What are some state of art computer vision models for anomaly detection that can learn continuously and build classes for detected anomalies?

I'm looking forward to build a model that: Detect anomalies Improve over user feedback Build classes for the anomalies based on user feedback Since a schema is worth a thousand words: Do you know ...
  • 111
0 votes
0 answers
13 views

When is predictive meaintenance feasible?

ML beginner here doing a proof of concept. I have a 1 column numeric dataset with pressure measurements of a machine, where I can observe multiple different kinds of strange sequences of values, some ...
0 votes
1 answer
52 views

Time series forecasting with non-temporal information (exogenous features)

I'm reviewing many time-series algorithms and libraries, such as Prophet, darts, auto_ts, etc. All libraries discuss univariant time-series (where the task is to forecast based on a single time-series)...
  • 113
0 votes
0 answers
35 views

Live peak / trough detection (data provided)

At the bottom of this question is the data of three time series in CSV-format. All are of same length and they all contain measurements of the same event "A". But each time series is using a ...
  • 101
0 votes
0 answers
26 views

Anomaly Detection with totally Categorical features

I am working on an anomaly detection project that aims to discover which merchant is the point of compromise. The data contains no numerical value and it looks like this; date account merchant fraud ...
0 votes
1 answer
252 views

Explanation of spectral residual algorithm for outlier detection

I've been reading the paper https://arxiv.org/pdf/1906.03821.pdf for spectral residual outlier detection, but I don't quite understand it. Specifically, in the implementation there are three variables ...
  • 11

1
2 3 4 5
7