Questions tagged [unsupervised-learning]

Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Ranking risky routes

i'm looking to get the top 10 rank of the most dangerous routes. I have a routes table where each row is a route and it has features such as avg daily traffic for past three years No. of times where ...
1 vote
3 answers
105 views

Real-Time Outlier/Anomaly Detection?

My data is the usage/playing statistics for players of a specific game. One data point for a user is aggregated statistics for one week. The goal is to be able to detect when the account of the player ...
1 vote
1 answer
89 views

An Unsupervised learning method suitable for large categorical data sets

I want to detect anomalies in the bank data set in an unsupervised learning method. However, in the bank data set, all columns except time and amount were categorical data, and about half of them had ...
0 votes
1 answer
193 views

Clustering 2D curves

I have a set of curves in 2D space each expressed as a set of (sampled) data points. Each set has more or less the same number of items - eventually I guess I’ll use binning to make sure the number of ...
2 votes
1 answer
447 views

Anomaly detection - relation between thresholds and anomalies

I'm developing an anomaly detection program in Python. Main idea is to create a new LSTM model every day, training it with the previous 7 days and predict the next day. Then, using thresholds, find ...
1 vote
2 answers
102 views

unsupervised anomaly detection for univariate fast frequency time series data?

I have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer This time series ...
1 vote
1 answer
127 views

Encode categorical data for unsupervised learning

What is the best encoder for categorical data in unsupervised learning? I am using unsupervised learning on mixed data (such as K-means). Before running my unsupervised algorithm, I am using dimension ...
1 vote
2 answers
114 views

approach for predicting machine failure using maintenance history

I have been struggling with this problem for a while now and I finally decided to post a question here to get some help. The problem i'm trying to solve is about predictive maintenance. Specifically, ...
1 vote
2 answers
138 views

How do I select the "best" unsupervised machine learning algorithm to cluster my specific dataset?

I want to cluster a dataset without prior knowledge on the correct amount of clusters. For different algorithms (i.e. k-means, gmm...) I can iterate through different values and try to find the best ...
0 votes
2 answers
193 views

customer segmentation with unbalanced data

I am trying to do a customer segmentation on my transactional data and I am struggling a little bit on the best approach. Since it is an unsupervised model I can throw it to any algorithm and get some ...
0 votes
2 answers
1k views

Percentile as a threshold for Anomaly Detection?

I'm following this article about Unsupervised Anomaly Detection Algorithms. In this article, a threshold value is calculated using the scipy score percentile method to determine whether the point is ...
1 vote
1 answer
76 views

Hierarchical dirichlet process results

I am thinking about using hierarchical dirichlet process to model a patent dataset. I've seen that HDP uses a base distribution and assumes that every topic comes from that base distribution. The ...
1 vote
1 answer
30 views

Removing specific phrases from textual data (R)

I have the following reddit posts and I would like to clean the posts and remove from the data the specific phrase "Click to expand", while keeping all other words within a post the same. <...
3 votes
1 answer
4k views

Anomaly detection using k-means clustering in Python

I'm working on an anomaly detection task in Python. Datasets regard a collection of time series coming from a sensor, so data are timestamps and the relative values. In order to find anomalies, I'm ...
0 votes
2 answers
502 views

How to group every data point with HDBSCAN to some group to have no noise?

TASK I am clustering products with about 70 dimensions ex.: price, rating 5/5, product tag(cleaning, toy, food, fruits) I use HDBSCAN to do it GOAL The goal is when users come on our site and I can ...
0 votes
1 answer
235 views

Clustering with custom criterion (minimum cluster weight)

Edit: following comment from @anony-mousse, I'm changing the question to search for a general clustering approach that matches this criterion (minimum weight per cluster). I am to use a clustering ...
0 votes
3 answers
1k views

What kind of learning is needed for anomaly detection? Supervised learning, semi-supervised learning or unsupervised learning?

I am doing anomaly detection recently, one of the methods is using AEs model to learn the pattern of normal samples. Determine it as an abnormal sample if it doesn’t match the pattern of normal ...
1 vote
2 answers
491 views

How can we perform STS (Semantic Textual Similarity) on unsupervised dataset using deep learning?

How do you implement STS(Semantic Textual Similarity) on an unlabelled dataset? The dataset column contains Unique_id, text1 (...
0 votes
1 answer
211 views

Rigth way to find Lorentzian distance between 2 point

Following this paper and this paper, I'm trying to implement the formula for the Lorentzian distance between 2 points (aka the distance between 2 points in Lorentzian space). I'll use this a the ...
1 vote
1 answer
328 views

More weightage to a categorical feature for an Autoencoder model

I am using autoencoder for anomaly detection. I don't have any labels already and so its unsupervised. If I have categorical variables, I usually one hot encode before giving it to the model. I would ...
1 vote
1 answer
2k views

How to set the Reconstruction error threshold for anomaly detection using autoencoders?

Hi I am doing anomaly detection using auto encoders.I have trained the model using 'Non Anomalous' values.Now when I give anomalous points as test data. What should be the Reconstruction error ...
0 votes
0 answers
12 views

Unsupervised rule extraction of categorical data

I have a dataset of network traffic with three features that I would like to extract rules from in order to apply firewall/flow control rules i.e. the permitted flows. I am able to classify a ...
2 votes
1 answer
527 views

How to tune / choose the preference parameter of AffinityPropagation?

I have large dictionary of "pairwise similarity matrixes" that would look like the following: similarity['group1']: ...
0 votes
1 answer
167 views

What methods are available to evaluate similarity between different clustering algorithms?

I am performing extensive customer segmentation analysis and so far implemented Gaussian Mixture Models, K-Means, and Hierarchical Clustering. For the most part, the algorithms agree on the structure ...
1 vote
1 answer
51 views

How to Justify Anomalies Detected by Unsupervised Anomaly Detection Models? [closed]

I'm working on an unsupervised anomaly detection project involving a large sensor dataset, where I aim to identify anomalies without the aid of labeled data. While I've implemented several ...
0 votes
1 answer
36 views

9-dimensional data classification

I have a dataset with 9 columns and 20 million rows. I have now split the dataset into two parts using IQR. One part is my inliers, the other part is my outliers. I now want to build a categorizer ...
1 vote
1 answer
455 views

How to approach Peak picking with a wide range of peak shapes, sizes, varying noise level, and occasionally shifting baseline?

I am trying write a program that continuously tracks the location a peak. To do that I need a very good peak detection algorithm. It not only has to tell the location of the peak but also the absence ...
3 votes
1 answer
88 views

Determine the most important documents for supervised learning

I have somewhat of a general/high level question. Assume I'm doing supervised machine learning on some text data (tweets for example) and categorizing the documents to a certain taxonomy (multi-class ...
1 vote
1 answer
1k views

Clustering on binary data

I am working on clustering on binary data which has 25 features, sample Feature 1 Feature 2 Feature 3 ...... Feature 25 1 1 0 0 011101 1 2 0 1 0 010011 0 3 1 0 1 101001 1 and I have used the ...
2 votes
1 answer
324 views

In Orange Data Mining, how do I use results from clustering a training-set to test and score a test-set?

I am performing analysis on the well-known 'Adult' data-set, available on UCI using Orange Data Mining. In a PhD thesis, Pelleg (2004; pg 79) uses unsupervised clustering of the prescribed training ...
0 votes
1 answer
28 views

Overfitting in the trained model

For my project on the classification problem of predicting churn customers, I trained various base models using k-fold validation on the training dataset and out of which random forest gave the best ...
0 votes
1 answer
136 views

Clustering cartesian coordinates associated with 1 categorical feature

I have a series of 2D coordinates X = {x, y}. Each are associated with one categorical variable W that can take 7 different values. E.g: ...
0 votes
1 answer
127 views

Stability of clusters in a unsupervised machine learning

I am new to Unsupervised learning. I am working on a customer segmentation data (with no labels). I have done K-Means and also calculated the silhouette score for the model. Now I want to study, if ...
1 vote
1 answer
160 views

K-Medoid Clustering with Point Weights

I asked the same question at Cross Validated, here I implemented a K-Medoid clustering algorithm recently; I have a number of points $x_1, ..., x_n$ which have various properties and a distance ...
3 votes
1 answer
275 views

Finding dominating attributes with in the clusters generated

I am having a dataset of customers where each customer is represented as some feature vector and I am applying K-means algorithm to this dataset. On the basis of those features, I can abstract and ...
0 votes
0 answers
11 views

How to update the weights of a ranking algorithms through unsupervised learning?

I have, let's say, 100 samples with n features, and I want to rank them by learning the weights for each feature. Weights must be learned from implicit ratings from the user (eg. user click). However, ...
3 votes
1 answer
574 views

What value can I gain by doing exploratory data analysis on features (and thus data) before doing clustering?

This might not be a very good question, but I would still ask if it's beneficial to do EDA before running a clustering algorithm? I understand that EDA helps us generate good and helpful insights ...
1 vote
3 answers
451 views

I have 32k black and white images. Want to do clustering on them

As the title says I'm trying to do clustering on a set of black and white images. These images are all 200x200 with black dots on a white canvas Example pics here (These are not actual photos from the ...
1 vote
1 answer
450 views

Clustering using both text and numerical features

I have a dataset that contains 2 types of features, one is generated from doc2vec and one is numerical feature. I would like to perform clustering analysis on them. However, due to the size of doc2vec ...
1 vote
1 answer
405 views

Time-series clustering Quality Measures

I am clustering time-series datasets which are not labeled (No Ground truth) and I want to measure the quality of the clusters. Could you please suggest any Clustering performance evaluation methods ...
0 votes
2 answers
346 views

Topic models for non-textual data?

I am looking to employ an unsupervised clustering on a dataset where each observation has a mix of textual and non-textual features. For each observation, I combine the features into a single vector ...
0 votes
1 answer
28 views

Different scaling methods of different features results in a faux dependency between them

My dataset contains the following two features: "movie duration" (minutes) and "tv shows duration" (seasons). If a certain sample is of type "movie", it's duration will ...
0 votes
1 answer
95 views

Word2vec to encode medical procedures when using isolation forests

I am planning to use Isolation Forests in R (solitude package) to identify outlier medical claims in my data. Each row of my data represents the group of drugs that each provider has administered in ...
3 votes
2 answers
748 views

Is it possible to make a label automatically in supervised learning(Machine Learning)?

My background knowledge: Basically, supervised learning is based on labeled data. Using the labeled data, the machine can study and determine results for unlabeled data. To do that, for example, if we ...
0 votes
1 answer
52 views

Clustering task: drop or not drop a categorical attribute/feature for which each row in the dataset contains a different value

I am dealing with a clustering task. In the dataset I am using there is a categorical feature and for each row in the dataset I have a different value for that feature (my dataset consists of 1000 ...
0 votes
0 answers
10 views

What kind of learning do I need ? (use-case specific)

Consider a scenario where I have a model trained on gesture videos (say a 3D ResNet). I am looking for a technique (or a combination) that allows me to further train the model every time I have a new ...
1 vote
1 answer
442 views

Difference between Q-learning and G-learning in Reinforcement Learning?

What is the difference between Q-learning and G-learning in Reinforcement Learning? Please explain with formulas. An example source: Instead of relying on a utility of consumption, we present G-...
0 votes
1 answer
659 views

Restricted Boltzmann Machine (RBM) implementation in Tensorflow (TF) 2.x

I‘m looking for a Python implementation of a Restricted Boltzmann Machine (RBM), e.g. applied to MNIST data as mentioned in „Elements of Statistical Learning“ Ch. 17, in Tensorflow 2.x. I‘m aware of ...
1 vote
1 answer
118 views

Classifying variable types on a list of variables

I have a list of around 700 variables which I need to perform a variable cleanup on. What complicates things is there are different numeric codes which flag an invalid value and these differ by the ...
1 vote
1 answer
831 views

Clustering for Sparse Data Matrix of high dimension

I currently have a dataset of 1000 entries with 512 features that are sparse. I want to cluster them. I have attempted using kmeans, but found that the clustering wasn't very good, and have been ...

1
2 3 4 5
10