Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

How to detect anomalies?

I have timeseries data with one value per day for a year. (there is one column with temperature data). I am using autoencoders to train a reconstruction model with mse loss. Firstly, I normalized the ...
user avatar
0 votes
0 answers
7 views

Which machine learning technique can be used for predictive log analysis

I have log data with 100k records. And These parameters. It looks like this. message types can be helpful for anomaly type detection. Out of total 15 message 5 ...
user avatar
0 votes
1 answer
13 views

How to find an anomalous matrix among many?

Let's say we have a bunch of matrices that we know are non-anomalous. We now receive a new matrix and want to know if it belongs into the group or is way off. Is there a way to do that? I'm thinking ...
user avatar
0 votes
0 answers
14 views

Unsupervised learning for anomaly detection based on memory and cpu usage

Recently got into Data Science, I've been working on a data science project, I have built a system that collects real-time logs on virtual machines in the cloud, the logs include memory consumption ...
user avatar
0 votes
1 answer
21 views

How can I generate anomalies in a dataset?

I'm building a tensorflow model to detect anomalies in an electricity smart meter data and I'm using UK-DALE Dataset. How can I introduce anomalies in the data so I can test the model?
user avatar
0 votes
0 answers
7 views

Confidence/prediction interval vs lower and upper bounds in ARIMA_PLUS big query

How is CI intervals in ML.forecast and lower-upper bounds in ML.Detect_Anomalies different for Arima_plus model in BigqueryML ?
user avatar
0 votes
1 answer
24 views

Training data for anomaly detection using LSTM Autoencoder

I am building an time-series anomaly detection engine using LSTM autoencoder. I read this article where the author suggests to train the model on clean data only in response to a comment. However, in ...
user avatar
0 votes
1 answer
17 views

regarding computing the centroid of high dimensional data

In scikit-learn, or other python libraries, are there any existing implementations to compute centroid for high dimensional data sets?
user avatar
1 vote
0 answers
13 views

Multiple Timeseries Anomaly Detection - identifying which feature is anomalous?

I have a multivariate time series where I have features such as: temperature set point energy used relative humidity, etc. Currently, I'm creating univariate anomaly detection models in Python using ...
user avatar
1 vote
1 answer
21 views

how to select threshold for unsupervised anomaly detection

I am working on an anomaly detection use case. I studied one technique of selecting the threshold that marks 5% of validation data as anomalies. how it works in anomaly detection cases. and there is ...
user avatar
  • 175
0 votes
0 answers
14 views

One anomaly detection model for all industries

Background - I'm creating a time-series anomaly detection (TSAD) model for the wifi throughput. My customers are 2 banks, 5 retail stores, 4 universities, 6 hospitals. Currently, I have 2 options to ...
user avatar
0 votes
0 answers
18 views

Anomaly detection and root cause analysis

ARIMA is widely used for anomaly detection on time-series data e.g. stock price prediction. ARIMA assumes that future value of a variable (stock price in our case) is dependent on its previous values. ...
user avatar
1 vote
1 answer
39 views

how to set threshold for anomaly detection

I read one research paper and they said that they are using a threshold for anomaly detection. The threshold is determined to make some proportion of data of the validation dataset labeled as ...
user avatar
  • 175
1 vote
0 answers
18 views

Fraud detection with machine learning with partialy labeled data

I am facing difficulties handling an issue with my dataset. The purpose of my analysis is to find an electricity thief given a dataset. The first dataset: contains the list of all customers (unlabeled,...
user avatar
  • 11
0 votes
0 answers
12 views

How to detect for certain malicious behavior?

I'm new to data science, and I'm trying to create a model that would help me detect certain malicious behaviors, e.g. beaconing traffic. Say I have looked at several types of beaconing traffic, and ...
user avatar
  • 111
0 votes
0 answers
27 views

Incremental learning on Autoencoder for anomaly detection

I want to incrementally train my pre-trained autoencoder model on data being received every minute. Based on this thread, successive calls to model.fit will incrementally train the model. However, the ...
user avatar
  • 11
0 votes
0 answers
8 views

Can I run isolation forest on existing data to find anomalies, save it for the future and use it on incoming data?

One of the major arguments I had recently is if we can save an unsupervised learning model to disk and use it later on incoming data. Isolation forest is one of the models that I use a lot for ...
user avatar
0 votes
1 answer
20 views

forcasting anomaly in products

I have a question about the forecasting of anomalies. I would be very grateful if you could refer me to some papers that deal with this kind of problem or give me some hints to start with this problem....
user avatar
  • 101
0 votes
0 answers
6 views

What kind of features can I obtain from IP:Port data?

I have a dataset that consist of the fields below. IP_Version,id,IP_TTL,IP_Source,TCP_Source_PORT,IP_Dest,TCP_Dest_PORT,data_size,timestamp What kind of features ...
user avatar
1 vote
0 answers
38 views

Can one use PCA to reduce the dimensionality of One-Hot-Encoded data?

I read a couple times that PCA was used as a method to reduce dimensionality for one-hot-encoded data. However, there were also some comments that using PCA is not a good idea since one-hot-encoded ...
user avatar
2 votes
1 answer
31 views

Various models giving 99% accuracy for KDDcup 99 dataset for Intrusion Detection, is there some sort of data leak I am missing?

Student who is quite new to all this here. I am currently working with the KDDcup 99 data for intrusion detection using various ML models (and ANN). My problem is that I am getting 99% often for ...
user avatar
1 vote
1 answer
26 views

Anomaly detection for varying dictionary

I want to detect the anomaly in the processes taking up the most CPU percent. I receive the data as a time series of dictionary values like so: ...
user avatar
  • 11
0 votes
0 answers
14 views

Modeling Approach for ID Persistence

I'm modeling a noisy dataset of ID's and have been tasked with creating a model to predict whether an ID exists at some arbitrary point in time. This is necessary, because an ID may not be observed at ...
user avatar
0 votes
1 answer
36 views

Word2vec to encode medical procedures when using isolation forests

I am planning to use Isolation Forests in R (solitude package) to identify outlier medical claims in my data. Each row of my data represents the group of drugs that each provider has administered in ...
user avatar
  • 231
0 votes
1 answer
26 views

What are some state of art computer vision models for anomaly detection that can learn continuously and build classes for detected anomalies?

I'm looking forward to build a model that: Detect anomalies Improve over user feedback Build classes for the anomalies based on user feedback Since a schema is worth a thousand words: Do you know ...
user avatar
  • 111
0 votes
0 answers
13 views

When is predictive meaintenance feasible?

ML beginner here doing a proof of concept. I have a 1 column numeric dataset with pressure measurements of a machine, where I can observe multiple different kinds of strange sequences of values, some ...
user avatar
0 votes
1 answer
33 views

Time series forecasting with non-temporal information (exogenous features)

I'm reviewing many time-series algorithms and libraries, such as Prophet, darts, auto_ts, etc. All libraries discuss univariant time-series (where the task is to forecast based on a single time-series)...
user avatar
  • 113
0 votes
0 answers
14 views

Live peak / trough detection (data provided)

At the bottom of this question is the data of three time series in CSV-format. All are of same length and they all contain measurements of the same event "A". But each time series is using a ...
user avatar
  • 101
0 votes
0 answers
21 views

Anomaly Detection with totally Categorical features

I am working on an anomaly detection project that aims to discover which merchant is the point of compromise. The data contains no numerical value and it looks like this; date account merchant fraud ...
user avatar
0 votes
1 answer
77 views

Explanation of spectral residual algorithm for outlier detection

I've been reading the paper https://arxiv.org/pdf/1906.03821.pdf for spectral residual outlier detection, but I don't quite understand it. Specifically, in the implementation there are three variables ...
user avatar
  • 11
0 votes
0 answers
17 views

Anomaly Detection in Discrete Sequences

So, I have a set of discrete sequences, let's say, composed by letters, representing a certain action: A -> B -> C -> D B -> E -> A -> C ... My ...
user avatar
  • 1
0 votes
1 answer
89 views

How to gps data anomaly detection in python

I have gps format dataset lat, lon. I want to detection anomaly using python. I tested knn, smv, cof, iforest using pycaret. But i did not. These colors anomlay because the angle change is too much ...
user avatar
  • 1
2 votes
1 answer
99 views

Find the Outlier

I have data that contains points (geo coordinates of a random planet, integer pairs) that represent places where land is definitely there. Here is an example with ...
user avatar
0 votes
0 answers
23 views

What type of Anomaly Detection Model could I use?

I would like to create an anomaly detection model that assigns a probability of risk instead of labels (1 or 0). My problem is that I only know for sure which records are anomalous but not which are ...
user avatar
0 votes
1 answer
31 views

Assign a risk score in records in a dataset

I was wondering, if I have a dataset with categorical and numerical data and labels such as 1 or 0 that shows if a row is anomalous or normal respectively. Is it possible to create somehow a model ...
user avatar
1 vote
0 answers
52 views

Nonparametric Outlier Detection

Which Nonparametric outlier detection do you suggest to detect outliers (red points) in these plots? I have tested std, IQR, etc., but no good result. It is just one vector including normal and ...
user avatar
  • 443
1 vote
1 answer
69 views

Anomaly detection and replacing it with past values in time series

I am trying to use anomaly detection to find the anomalies in my time series, and if I find it, I will replace it with my past values. I'm trying to do this because I want to create an upper and lower ...
user avatar
0 votes
1 answer
18 views

Anomaly Detection Techniques

Often the hardest part of solving an Anomaly Detection problem can be finding the right technique for the job. Different Anomaly techniques are better suited for different types of data and different ...
user avatar
  • 2,301
0 votes
1 answer
53 views

how to set threshold value by looking at loss distribution in anomaly detection task

I am following this tutorial https://towardsdatascience.com/lstm-autoencoder-for-anomaly-detection-e1f4f2ee7ccf to use LSTM autoencoder to detect anomalies in my unsupervised dataset. they plotted ...
user avatar
  • 175
0 votes
0 answers
55 views

How to prepere dataset for binary classification (anomaly detection?) on timestamped sensor data (multiple files)?

my goal is to make prediction (good or bad data) on sensor data. I tried a lot, but failed to shape my data to get the desired output. scenario: I have multiple timestamped (time as it self is not ...
user avatar
  • 9
0 votes
0 answers
16 views

Generation of Anomalous data points

I have a task to generate some anomalous points in a real world dataset with 15 features, and a synthetic dataset of 5 features. I was thinking of using correlation between features, but it'll be a ...
user avatar
1 vote
1 answer
95 views

How to detect anomalies in each feature - time series

I have a dataset with 5 features corresponding to 5 sensors that measure each three seconds the state of an accelerator. It is structured as well: ...
user avatar
  • 43
0 votes
0 answers
19 views

Regression followed by thresholding to predict rare events

I have a multi-variate time series for which I am performing forecasting by regression. My aim is to forecast extreme values in this time series (rare events). On the one hand I have a regression ...
user avatar
  • 121
0 votes
0 answers
9 views

What is the name of this technique involving tracking cumulative errors with a forgiveness parameter?

I'm looking for the name of a technique I've seen used before. Most common in time-series based anomaly detection. It involves keeping a running total of consecutive "error" amounts, ...
user avatar
  • 101
0 votes
0 answers
16 views

Clustering 3D image voxels based on their location and value

My goal is to detect whether a MRI image contains an anomaly and the location of the anomaly. In my dataset I have MRI brain images which contain values of electrical conductivity of brain tissues. ...
user avatar
0 votes
0 answers
45 views

How to evaluate unsupervised Anomaly Detection using k-means

I'm trying out different anomaly detection models and would love to hear opinion on my idea from somebody experienced. My goal is to perform anomaly detection with different models and to give each ...
user avatar
  • 1
1 vote
2 answers
55 views

How do I determine the top "reason" for anomaly when using Isolation Forests

I am using Isolation Forests for Anomaly Detection. Say, my set has 10 variables, var1, var2, ..., var10, and I found an anomaly. Can I rank the 10 variables var1, var2, ..., var10 in such a way I can ...
user avatar
  • 1,851
0 votes
1 answer
39 views

which statistical parameters are more useful to detect anomalies and outlier? mean max min var?

This time series contains some time frame which each of them are 8K (frequencies)*151 (time samples) in 0.5 sec [overall 1.2288 millions samples per half a second) I need to find anomalous based on ...
user avatar
  • 1,516
0 votes
2 answers
291 views

Anomaly (Outlier) Detection with Isolation Forest too sensitive even with low contamination

I'm trying to use the sklearn implementation of the Isolation Forest algorithm to detect anomalies in my time series data. However, even with a very low contamination parameter (0.0001), it is ...
user avatar
0 votes
0 answers
23 views

What is the range of Anomaly_Score column when using PyCaret anomaly detection library?

When we use the PyCaret library for anomaly detection we import it using from pycaret.anomaly import * and we create a model using the ...
user avatar
  • 135

1
2 3 4 5
7