Questions tagged [unsupervised-learning]
Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.
451
questions
0
votes
0
answers
12
views
Ranking risky routes
i'm looking to get the top 10 rank of the most dangerous routes. I have a routes table where each row is a route and it has features such as
avg daily traffic for past three years
No. of times where ...
- 1
1
vote
3
answers
105
views
Real-Time Outlier/Anomaly Detection?
My data is the usage/playing statistics for players of a specific game. One data point for a user is aggregated statistics for one week. The goal is to be able to detect when the account of the player ...
CommunityBot
- 1
1
vote
1
answer
89
views
An Unsupervised learning method suitable for large categorical data sets
I want to detect anomalies in the bank data set in an unsupervised learning method. However, in the bank data set, all columns except time and amount were categorical data, and about half of them had ...
CommunityBot
- 1
0
votes
1
answer
193
views
Clustering 2D curves
I have a set of curves in 2D space each expressed as a set of (sampled) data points.
Each set has more or less the same number of items - eventually I guess I’ll use binning to make sure the number of ...
CommunityBot
- 1
2
votes
1
answer
447
views
Anomaly detection - relation between thresholds and anomalies
I'm developing an anomaly detection program in Python.
Main idea is to create a new LSTM model every day, training it with the previous 7 days and predict the next day.
Then, using thresholds, find ...
CommunityBot
- 1
1
vote
2
answers
102
views
unsupervised anomaly detection for univariate fast frequency time series data?
I have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer
This time series ...
CommunityBot
- 1
1
vote
1
answer
127
views
Encode categorical data for unsupervised learning
What is the best encoder for categorical data in unsupervised learning?
I am using unsupervised learning on mixed data (such as K-means).
Before running my unsupervised algorithm, I am using dimension ...
CommunityBot
- 1
1
vote
2
answers
114
views
approach for predicting machine failure using maintenance history
I have been struggling with this problem for a while now and I finally decided to post a question here to get some help. The problem i'm trying to solve is about predictive maintenance.
Specifically, ...
CommunityBot
- 1
1
vote
2
answers
138
views
How do I select the "best" unsupervised machine learning algorithm to cluster my specific dataset?
I want to cluster a dataset without prior knowledge on the correct amount of clusters. For different algorithms (i.e. k-means, gmm...) I can iterate through different values and try to find the best ...
CommunityBot
- 1
0
votes
2
answers
193
views
customer segmentation with unbalanced data
I am trying to do a customer segmentation on my transactional data and I am struggling a little bit on the best approach. Since it is an unsupervised model I can throw it to any algorithm and get some ...
CommunityBot
- 1
0
votes
2
answers
1k
views
Percentile as a threshold for Anomaly Detection?
I'm following this article about Unsupervised Anomaly Detection Algorithms. In this article, a threshold value is calculated using the scipy score percentile method to determine whether the point is ...
CommunityBot
- 1
1
vote
1
answer
76
views
Hierarchical dirichlet process results
I am thinking about using hierarchical dirichlet process to model a patent dataset. I've seen that HDP uses a base distribution and assumes that every topic comes from that base distribution.
The ...
CommunityBot
- 1
1
vote
1
answer
30
views
Removing specific phrases from textual data (R)
I have the following reddit posts and I would like to clean the posts and remove from the data the specific phrase "Click to expand", while keeping all other words within a post the same.
<...
3
votes
1
answer
4k
views
Anomaly detection using k-means clustering in Python
I'm working on an anomaly detection task in Python.
Datasets regard a collection of time series coming from a sensor, so data are timestamps and the relative values.
In order to find anomalies, I'm ...
CommunityBot
- 1
0
votes
2
answers
502
views
How to group every data point with HDBSCAN to some group to have no noise?
TASK
I am clustering products with about 70 dimensions ex.: price, rating 5/5, product tag(cleaning, toy, food, fruits)
I use HDBSCAN to do it
GOAL
The goal is when users come on our site and I can ...
CommunityBot
- 1
0
votes
1
answer
235
views
Clustering with custom criterion (minimum cluster weight)
Edit: following comment from @anony-mousse, I'm changing the question to search for a general clustering approach that matches this criterion (minimum weight per cluster).
I am to use a clustering ...
CommunityBot
- 1
0
votes
3
answers
1k
views
What kind of learning is needed for anomaly detection? Supervised learning, semi-supervised learning or unsupervised learning?
I am doing anomaly detection recently, one of the methods is using AEs model to learn the pattern of normal samples.
Determine it as an abnormal sample if it doesn’t match the pattern of normal ...
CommunityBot
- 1
1
vote
2
answers
491
views
How can we perform STS (Semantic Textual Similarity) on unsupervised dataset using deep learning?
How do you implement STS(Semantic Textual Similarity) on an unlabelled dataset? The dataset column contains Unique_id, text1 (...
CommunityBot
- 1
0
votes
1
answer
211
views
Rigth way to find Lorentzian distance between 2 point
Following this paper and this paper, I'm trying to implement the formula for the Lorentzian distance between 2 points (aka the distance between 2 points in Lorentzian space). I'll use this a the ...
CommunityBot
- 1
1
vote
1
answer
328
views
More weightage to a categorical feature for an Autoencoder model
I am using autoencoder for anomaly detection. I don't have any labels already and so its unsupervised. If I have categorical variables, I usually one hot encode before giving it to the model. I would ...
CommunityBot
- 1
1
vote
1
answer
2k
views
How to set the Reconstruction error threshold for anomaly detection using autoencoders?
Hi I am doing anomaly detection using auto encoders.I have trained the model using 'Non Anomalous' values.Now when I give anomalous points as test data.
What should be the Reconstruction error ...
CommunityBot
- 1
0
votes
0
answers
12
views
Unsupervised rule extraction of categorical data
I have a dataset of network traffic with three features that I would like to extract rules from in order to apply firewall/flow control rules i.e. the permitted flows.
I am able to classify a ...
2
votes
1
answer
527
views
How to tune / choose the preference parameter of AffinityPropagation?
I have large dictionary of "pairwise similarity matrixes" that would look like the following:
similarity['group1']:
...
CommunityBot
- 1
0
votes
1
answer
167
views
What methods are available to evaluate similarity between different clustering algorithms?
I am performing extensive customer segmentation analysis and so far implemented Gaussian Mixture Models, K-Means, and Hierarchical Clustering. For the most part, the algorithms agree on the structure ...
CommunityBot
- 1
1
vote
1
answer
51
views
How to Justify Anomalies Detected by Unsupervised Anomaly Detection Models? [closed]
I'm working on an unsupervised anomaly detection project involving a large sensor dataset, where I aim to identify anomalies without the aid of labeled data. While I've implemented several ...
- 1,205
0
votes
1
answer
36
views
9-dimensional data classification
I have a dataset with 9 columns and 20 million rows. I have now split the dataset into two parts using IQR.
One part is my inliers, the other part is my outliers.
I now want to build a categorizer ...
- 1,918
1
vote
1
answer
455
views
How to approach Peak picking with a wide range of peak shapes, sizes, varying noise level, and occasionally shifting baseline?
I am trying write a program that continuously tracks the location a peak. To do that I need a very good peak detection algorithm. It not only has to tell the location of the peak but also the absence ...
CommunityBot
- 1
3
votes
1
answer
88
views
Determine the most important documents for supervised learning
I have somewhat of a general/high level question.
Assume I'm doing supervised machine learning on some text data (tweets for example) and categorizing the documents to a certain taxonomy (multi-class ...
CommunityBot
- 1
1
vote
1
answer
1k
views
Clustering on binary data
I am working on clustering on binary data which has 25 features,
sample
Feature 1
Feature 2
Feature 3
......
Feature 25
1
1
0
0
011101
1
2
0
1
0
010011
0
3
1
0
1
101001
1
and I have used the ...
CommunityBot
- 1
2
votes
1
answer
324
views
In Orange Data Mining, how do I use results from clustering a training-set to test and score a test-set?
I am performing analysis on the well-known 'Adult' data-set, available on UCI using Orange Data Mining. In a PhD thesis, Pelleg (2004; pg 79) uses unsupervised clustering of the prescribed training ...
CommunityBot
- 1
0
votes
1
answer
28
views
Overfitting in the trained model
For my project on the classification problem of predicting churn customers, I trained various base models using k-fold validation on the training dataset and out of which random forest gave the best ...
- 120
0
votes
1
answer
136
views
Clustering cartesian coordinates associated with 1 categorical feature
I have a series of 2D coordinates X = {x, y}. Each are associated with one categorical variable W that can take 7 different values.
E.g:
...
CommunityBot
- 1
0
votes
1
answer
127
views
Stability of clusters in a unsupervised machine learning
I am new to Unsupervised learning. I am working on a customer segmentation data (with no labels). I have done K-Means and also calculated the silhouette score for the model. Now I want to study, if ...
CommunityBot
- 1
1
vote
1
answer
160
views
K-Medoid Clustering with Point Weights
I asked the same question at Cross Validated, here
I implemented a K-Medoid clustering algorithm recently; I have a number of points $x_1, ..., x_n$ which have various properties and a distance ...
CommunityBot
- 1
3
votes
1
answer
275
views
Finding dominating attributes with in the clusters generated
I am having a dataset of customers where each customer is represented as some feature vector and I am applying K-means algorithm to this dataset. On the basis of those features, I can abstract and ...
CommunityBot
- 1
0
votes
0
answers
11
views
How to update the weights of a ranking algorithms through unsupervised learning?
I have, let's say, 100 samples with n features, and I want to rank them by learning the weights for each feature. Weights must be learned from implicit ratings from the user (eg. user click). However, ...
3
votes
1
answer
574
views
What value can I gain by doing exploratory data analysis on features (and thus data) before doing clustering?
This might not be a very good question, but I would still ask if it's beneficial to do EDA before running a clustering algorithm?
I understand that EDA helps us generate good and helpful insights ...
CommunityBot
- 1
1
vote
3
answers
451
views
I have 32k black and white images. Want to do clustering on them
As the title says I'm trying to do clustering on a set of black and white images. These images are all 200x200 with black dots on a white canvas Example pics here (These are not actual photos from the ...
CommunityBot
- 1
1
vote
1
answer
450
views
Clustering using both text and numerical features
I have a dataset that contains 2 types of features, one is generated from doc2vec and one is numerical feature. I would like to perform clustering analysis on them. However, due to the size of doc2vec ...
CommunityBot
- 1
1
vote
1
answer
405
views
Time-series clustering Quality Measures
I am clustering time-series datasets which are not labeled (No Ground truth) and I want to measure the quality of the clusters.
Could you please suggest any Clustering performance evaluation methods ...
CommunityBot
- 1
0
votes
2
answers
346
views
Topic models for non-textual data?
I am looking to employ an unsupervised clustering on a dataset where each observation has a mix of textual and non-textual features.
For each observation, I combine the features into a single vector ...
CommunityBot
- 1
0
votes
1
answer
28
views
Different scaling methods of different features results in a faux dependency between them
My dataset contains the following two features: "movie duration" (minutes) and "tv shows duration" (seasons). If a certain sample is of type "movie", it's duration will ...
- 1,167
0
votes
1
answer
95
views
Word2vec to encode medical procedures when using isolation forests
I am planning to use Isolation Forests in R (solitude package) to identify outlier medical claims in my data.
Each row of my data represents the group of drugs that each provider has administered in ...
CommunityBot
- 1
3
votes
2
answers
748
views
Is it possible to make a label automatically in supervised learning(Machine Learning)?
My background knowledge: Basically, supervised learning is based on labeled data. Using the labeled data, the machine can study and determine results for unlabeled data. To do that, for example, if we ...
CommunityBot
- 1
0
votes
1
answer
52
views
Clustering task: drop or not drop a categorical attribute/feature for which each row in the dataset contains a different value
I am dealing with a clustering task. In the dataset I am using there is a categorical feature and for each row in the dataset I have a different value for that feature (my dataset consists of 1000 ...
- 98
0
votes
0
answers
10
views
What kind of learning do I need ? (use-case specific)
Consider a scenario where I have a model trained on gesture videos (say a 3D ResNet). I am looking for a technique (or a combination) that allows me to further train the model every time I have a new ...
- 139
1
vote
1
answer
442
views
Difference between Q-learning and G-learning in Reinforcement Learning?
What is the difference between Q-learning and G-learning in Reinforcement Learning? Please explain with formulas.
An example source:
Instead of relying on a utility of consumption, we present G-...
CommunityBot
- 1
0
votes
1
answer
659
views
Restricted Boltzmann Machine (RBM) implementation in Tensorflow (TF) 2.x
I‘m looking for a Python implementation of a Restricted Boltzmann Machine (RBM), e.g. applied to MNIST data as mentioned in „Elements of Statistical Learning“ Ch. 17, in Tensorflow 2.x.
I‘m aware of ...
CommunityBot
- 1
1
vote
1
answer
118
views
Classifying variable types on a list of variables
I have a list of around 700 variables which I need to perform a variable cleanup on. What complicates things is there are different numeric codes which flag an invalid value and these differ by the ...
CommunityBot
- 1
1
vote
1
answer
831
views
Clustering for Sparse Data Matrix of high dimension
I currently have a dataset of 1000 entries with 512 features that are sparse. I want to cluster them. I have attempted using kmeans, but found that the clustering wasn't very good, and have been ...
CommunityBot
- 1