Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

data science ideas for payment integrity in healthcare insurance

I am looking for new data science ideas for payment integrity (waste, fraud abuse) in healthcare insurance. Can you suggests some articles, links, blogs about metrics or ml techniques?
0
votes
0answers
11 views

Anomaly Detection in Highly Variable Time-Series Data

I am trying to detect anomalies through a column called count. The data is a time-series data and it is present for every 5 minutes for each day. The dataframe looks like this: ...
1
vote
0answers
36 views

Anomaly detection for high dimensional categorical data

I have a dataset with around 200+ categorical variables $X_i$ and the sizes of their domain $|X_i|$ range from 2 to 8k. So, if I one-hot encode the combination of these variables, the vector space (...
1
vote
0answers
21 views

Interpreting one-class SVM

I am new to SVM (one-class) and was practically investigating it. Got some weird result that I can not explain. Let me demonstrate by some small reproducible code and visualization: ...
0
votes
0answers
7 views

How can interparet shap.summary_plot and its gray color concerning outliers/anomaly?

I inspired by this notebook, and I'm experimenting IsolationForest algorithm using scikit-learn==0.22.2.post1 for anomaly ...
1
vote
1answer
55 views

Similarity between 2 statistical distributions

Is there any index that measures similarity between 2 gaussian distributions of 1-D data (may have slightly different number of points) considering their mean shift, variance shift, difference in ...
1
vote
0answers
17 views

How to calculate MAE and threshold in a multivariate time series

I'm trying to understand how to calculate the MAE in my time series and then the thresholds to understand which of my data in the test set are anomalies. I'm following this tutorial, which is based on ...
1
vote
2answers
99 views

Autenocoder and anomaly detection task

I'm trying to create an autoencoder for the anomaly detection task, but I'm noticing that even if it performs very well on the training set, it starts to stop recreating half of the test set. I tried ...
0
votes
1answer
32 views

Understanding time series anomaly detection using Autoencoder

I'm studying how to detect anomalies in the time series using an Autoeconder. In particular, I'm following the guide posted in the Keras website, but I don't understand why they are creating and how ...
0
votes
1answer
41 views

Anomaly Detection

I have a problem where I want to identify Vendors with unusual high amount invoices. What would be the best way to identify such invoices? I am trying to use Isolation Forest but having trouble in ...
1
vote
0answers
14 views

Finding optimal time series using convolution [closed]

we logged sensor data while milling a workpiece. At several points, the workpiece was damaged and this induced a certain sensor data time series. Due to noise and since its a real world measurement, ...
1
vote
1answer
53 views

Anomaly detection on sparse categorical data

I have a big dataset with a column "clientid" and a categorical column "choice". I want to find out what are the clients that have strange combinations of choices (less frequent ...
0
votes
0answers
29 views

Convolutional AE always overfitting time series - what’s wrong?

I've build a CAE for anomaly detection in time series, but it is always overfitting. I've tried data augmentation, short/long inputvector, dropout rates... I don't know what I'm doing wrong, may be ...
1
vote
0answers
16 views

How to preprocess data for multiple rows instance? [closed]

I have a machine learning problem where I must detect anomalies in (let say) taxes declaration. So I have multiple rows for each enterprise, each row describing the tax declaration a the time (date,...
1
vote
1answer
34 views

Geolocation Based Anomaly Detection in IPs Using Isolation Forest

I'm trying to detect anomalies based on geolocation from IP addresses on a server access log file. I have created two features country and geo_velocity, using the IP address and the timestamp of each ...
0
votes
1answer
25 views

Performing anomalie detection on a battery volatge using LSTM-RNN

I am trying to detect anomalies in a battery output voltage for one month. I have the next data frame, as it is shown the data is collected each minute for each day so I have almost 1420 sample per ...
0
votes
1answer
14 views

(Labeled, if possible) time-series datasets for anomaly detection [closed]

I would like to create a big list of available time-series datasets for anomaly detection. I'm especially interested in the following: The time-series data should be segmented into cycles Ideally, ...
0
votes
1answer
40 views

How to find anomalies in (almost) constant stream of data?

I have a process that (simply put), starts every 5 minutes, collects data, and put that data into the database. More detailed explanation would be that process starts, collects data (which takes some ...
3
votes
1answer
42 views

How to determine the abnormality of a specific variable by taking into account all the other variables in the data?

I have an issue of machine learning/anomaly detection. Indeed, I have a variable Y and several other variables X. The purpose is to quantify the degree of abnormality of the data on Y but I have to ...
0
votes
0answers
17 views

How to do parameter estimation on relatively few samples?

Context: I have a time series on which I've trained an autoencoder for anomaly detection (AD) purposes. My time-series is 6 years worth of daily data, but for AD purposes, I am looking at one week ...
0
votes
0answers
34 views

outliers in time series

I have a data set like the following where the rows are dates and the columns are values recorded by different sensors on those dates. Before working with the data for the purpose of predicting it, I ...
0
votes
0answers
48 views

How can realize the evaluation/validation of unsupervised models through unlabeled data?

I'm researching anomaly detection, which is nothing else than outliers detection on a set of time-series web servers access log data or network traffic. Recently I re-faced to following fundamental ...
1
vote
1answer
38 views

Labels as features in anomaly detection

I have a dataset born to solve a classification problem. Due to the imbalances of the Y, i choose to move to an anomaly detection task. Should I use the Y i have inside the anomaly detection model as ...
0
votes
1answer
46 views

Decision trees for anomaly detection

Problem From what I understand, a common method in anomaly detection consists in building a predictive model trained on non-anomalous training data, and perform anomaly detection using the error of ...
1
vote
0answers
14 views

Analysing wrongly predicted labels and pick most significant [closed]

I try to do behavior analysis of network systems based on network flow. I have a model that predicts if a certain flow pattern is a Windows Server, LX Server or Windows Client. (Highly simplified) I ...
2
votes
2answers
35 views

How to perform Anomaly Detection on a force profile?

I have a set of force profiles of an industrial machine. I'm trying to develop an algorithm that tries to understand when a new profile is "anomalous" with respect to the ones in "...
1
vote
1answer
154 views

unsupervised anomaly detection on sparse data

Given that I have a very sparse data matrix with continuous features, like this dataframe for example ...
0
votes
0answers
66 views

How to find a 'similarity' measure between 2 pandas dataframes?

I have 2 pandas dataframes: one big (300.000+ rows) and one little (50- rows) with the same columns. Assuming that the entries of the little one form a cluster I want to identify the entries of the ...
1
vote
1answer
98 views

K-Prototype for anomaly detection

I have logs of the form (e.g. from a gym login).. the representational case is so: UserName, Login time, timeSpend_on_weights, time_spent_on_elliptical ...
1
vote
0answers
22 views

Explanation of Excess Mass(EM)

I was researching on evaluation metrics to understand the performance of unsupervised anomaly detection algorithms and I came across this paper The author suggests that EM and MV based numerical ...
10
votes
1answer
577 views

Learning with Positive labels only

I have ~7 million rows of customer data (~500 sparse attributes) A million out of them have opted in to a new service. How do I use this signal to predict which of the remaining customers are likely ...
1
vote
1answer
39 views

Is window based sequencing a good idea to obtain more training data for LSTMs?

I am trying to do an unsupervised autoencoder based outlier detection for time series using LSTMs. Here, there are multiple time series, and an entire series is to be considered as an outlier. However,...
2
votes
2answers
40 views

Log analysis dataset with labeled cybersecurity issues

I am seeking to find a dataset with log files that have labeled cybersecurity issues. As I am trying to build a cybersecurity log analysis model there is no preference on the type of the log, but ...
0
votes
0answers
18 views

Can I treat concept drift detection as a binary clarification problem?

Assume I have the ground truth labels for both non-drited and drifted samples, can I treat concept drift detection as a binary classication problem (one class non-drifted one class drifted)? If not ...
0
votes
2answers
39 views

Looking for spatial clusters and anomalies. Is DBSCAN the right tool?

I have a regular 2D grid of data points (X, Y) with each point having a value. I'd like to identify clusters and then anomalies that don't belong to those clusters. I'm trying to understand the best ...
2
votes
1answer
37 views

Word representation that gives more weight to terms frequent in corpus?

The tf-idf discounts the words that appear in a lot of documents in the corpus. I am constructing an anomaly detection text classification algorithm that is trained only on valid documents. Later I ...
0
votes
1answer
147 views

Algorithms for Anomaly Detection of Event Sequence Data [Python/R]

I am building an anomaly detection system of event sequence data (transactions). For each timestep, a transaction can be in any of 76 different stages. My dataset is therefore a 3D array of size(m,t,N)...
0
votes
0answers
16 views

Identifying anomalies in spatial (latitude, longitude, time) data? [duplicate]

I'll start off by prefacing this with the fact that I'm not even sure if I'm asking the right question and what I'm really looking for is some guidance to dig in the right direction. I have a set of ...
0
votes
1answer
39 views

Incorrect multi-variate anomaly detection - Isolation Forest Python

My data looks like below. it has 333 rows and 2 columns. Clearly the first row is anomaly. ndf: ...
0
votes
0answers
25 views

Is there any inductive Graph Variational Auto Encoder?

I have been reading about how we can model a Variational AutoEncoder (VAE) into a Graph Variational AutoEncoder (GVAE) where the decoder reconstructs the adjacency matrix. I presume that this approach ...
0
votes
2answers
191 views

Cross-Validation in Anomaly Detection with Labelled Data

I am working on a project where I train anomaly detection algorithms Isolation Forest and Auto-Encoder. My data is labelled so I have the ground truth but the nature of the problem requires ...
1
vote
0answers
28 views

Multivariate Gaussian distribution - Covariance vs linear dependence

From prof. Andrew Ng's Multivariate Gaussian distribution lecture, covariance measures linear dependency between features, in which case we might use Multivariate Gaussian distribution with covariance ...
0
votes
0answers
89 views

Anomaly Detection over multivariate data containing Nominal and numerical predictors

I am trying to implement Anomaly Detection over a multivariate dataset having nominal and numerical predictors. Dataset has following pattern: If we consider the below sample records, category_id, ...
-1
votes
1answer
31 views

Does Anomaly Detection Algorithm works when the features are not correlated?

I am working on an Anomaly Detection Problem and the algorithm I used is an Autoencoder Multivariate Gaussian. The problem with my data is that it is unlabeled and not correlated. For example, let's ...
4
votes
1answer
542 views

Interpretation of scikit-learn one class svm scores

How can I interpret the scores generated by the function score_samples(X) from a scikit-learn OneClassSVM model? Is there a way ...
0
votes
1answer
82 views

Semi-supervised anomaly detection

I am currently exploring anomaly detection methods for my work and, basically I have gone through Local Oulier Factor and Isolation Forests, both unsupervised methods. Now, the thing is, there might ...
1
vote
2answers
47 views

Identify the parameter causing the anomaly in a multivariate dataset

I have a payment transaction dataset with a large number of predictor variables. I am trying to build a model for anomaly detection and I have evaluated various algorithms/approaches for the same like ...
2
votes
0answers
34 views

What's the best way to validate a rare event detection model during training?

When training a deep model for rare event detection (e.g. sound of an alarm in a home device audio stream), is it best to use a balanced validation set (50% alarm, 50% normal) to determine early ...
1
vote
0answers
181 views

Hyperparameter tuning one-class svm

I have a problem where I am trying to apply a one-class svm to detect outliers. I am training on a dataset of true cases using a one-class radial svm and then predicting for both false and true cases. ...

1
2 3 4 5 6