Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [unsupervised-learning]

Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.

Filter by
Sorted by
Tagged with
1 vote
1 answer
54 views

What is the difference between K-Means & Self Organized Maps?

It seems they both perform clustering. They both reduce the dimensionality of the input data and classify further inputs based upon their distance/similarity to the center points. These points then ...
1 vote
2 answers
280 views

How do I select the "best" unsupervised machine learning algorithm to cluster my specific dataset?

I want to cluster a dataset without prior knowledge on the correct amount of clusters. For different algorithms (i.e. k-means, gmm...) I can iterate through different values and try to find the best ...
0 votes
2 answers
224 views

customer segmentation with unbalanced data

I am trying to do a customer segmentation on my transactional data and I am struggling a little bit on the best approach. Since it is an unsupervised model I can throw it to any algorithm and get some ...
1 vote
1 answer
77 views

How to cluster text-based software requirements

I'm beginner in deep learning and I'd like to cluster text-based software requirements by themes (words similarities/frequency of words) using neural networks. Is there any example/tutorial/github ...
0 votes
2 answers
2k views

Percentile as a threshold for Anomaly Detection?

I'm following this article about Unsupervised Anomaly Detection Algorithms. In this article, a threshold value is calculated using the scipy score percentile method to determine whether the point is ...
2 votes
3 answers
2k views

How to create clusters based on sentence similarity?

I have data which looks like following. Data is a group of sentences which are similar, but have few unique words in between like TABLEA, TABLEB etc. ...
1 vote
1 answer
87 views

Hierarchical dirichlet process results

I am thinking about using hierarchical dirichlet process to model a patent dataset. I've seen that HDP uses a base distribution and assumes that every topic comes from that base distribution. The ...
0 votes
1 answer
262 views

Clustering with custom criterion (minimum cluster weight)

Edit: following comment from @anony-mousse, I'm changing the question to search for a general clustering approach that matches this criterion (minimum weight per cluster). I am to use a clustering ...
0 votes
0 answers
12 views

What to put for X_train, y_train when using it for unsupervised LSTM for anomaly detection?

I have a dataset with 5 features (excluding the date) [Result, Ward, Age, Facility, Resource] . The train dataset has non-anomalous data, and the test dataset will have some anomalous data. This ...
1 vote
2 answers
509 views

How can we perform STS (Semantic Textual Similarity) on unsupervised dataset using deep learning?

How do you implement STS(Semantic Textual Similarity) on an unlabelled dataset? The dataset column contains Unique_id, text1 (...
0 votes
3 answers
1k views

What kind of learning is needed for anomaly detection? Supervised learning, semi-supervised learning or unsupervised learning?

I am doing anomaly detection recently, one of the methods is using AEs model to learn the pattern of normal samples. Determine it as an abnormal sample if it doesn’t match the pattern of normal ...
3 votes
2 answers
620 views

What value can I gain by doing exploratory data analysis on features (and thus data) before doing clustering?

This might not be a very good question, but I would still ask if it's beneficial to do EDA before running a clustering algorithm? I understand that EDA helps us generate good and helpful insights ...
7 votes
4 answers
24k views

Anomaly detection on time series

I've just started working on an anomaly detection development in Python. My data sets are a collection of timeseries. More in details, data are coming from some sensors/meters which record and ...
2 votes
1 answer
58 views

What's the good index to choose number of clusters so that obtained clusters are homogeneous?

I perform a clustering on one-dimensional dataset and I need a way to automatically decide what's the optimal number of clusters from $k \in \{2, 3, 4, 5, 6\}$. The number of observations to cluster ...
0 votes
1 answer
729 views

Rigth way to find Lorentzian distance between 2 point

Following this paper and this paper, I'm trying to implement the formula for the Lorentzian distance between 2 points (aka the distance between 2 points in Lorentzian space). I'll use this a the ...
1 vote
1 answer
59 views

Identifying potential customers based on their Rank and Value

I have a dataset which has demographic data available for a list of new customers. the data does'nt include transaction data of the customers. I want to identify the top 100 potential customers among ...
1 vote
1 answer
338 views

More weightage to a categorical feature for an Autoencoder model

I am using autoencoder for anomaly detection. I don't have any labels already and so its unsupervised. If I have categorical variables, I usually one hot encode before giving it to the model. I would ...
2 votes
1 answer
651 views

How to tune / choose the preference parameter of AffinityPropagation?

I have large dictionary of "pairwise similarity matrixes" that would look like the following: similarity['group1']: ...
0 votes
0 answers
6 views

How to Identify Equipment Churn from Laboratory Service Records Without Direct Churn Labels?

I'm analyzing a dataset encompassing 20 years of laboratory equipment service records, which includes the equipment ID, service dates, types of equipment (HOOD_TYPE), and descriptions of performed ...
0 votes
1 answer
186 views

What methods are available to evaluate similarity between different clustering algorithms?

I am performing extensive customer segmentation analysis and so far implemented Gaussian Mixture Models, K-Means, and Hierarchical Clustering. For the most part, the algorithms agree on the structure ...
1 vote
1 answer
2k views

How to set the Reconstruction error threshold for anomaly detection using autoencoders?

Hi I am doing anomaly detection using auto encoders.I have trained the model using 'Non Anomalous' values.Now when I give anomalous points as test data. What should be the Reconstruction error ...
2 votes
1 answer
61 views

Tiering after clustering with Kmeans

I would like to have some suggestions on possible avenues that would make sense in the following context. 3 Optimal clusters have been identified in a 5000 list of customers using Kmeans Data model ...
3 votes
1 answer
105 views

Determine the most important documents for supervised learning

I have somewhat of a general/high level question. Assume I'm doing supervised machine learning on some text data (tweets for example) and categorizing the documents to a certain taxonomy (multi-class ...
1 vote
1 answer
61 views

Clustering with hierarchical data dependencies

I am currently looking into how to cluster data with hierarchical dependencies. An example of a problem that I want to cluster: we would like to cluster cities to identify similar characteristics with ...
2 votes
1 answer
1k views

Clustering on binary data

I am working on clustering on binary data which has 25 features, sample Feature 1 Feature 2 Feature 3 ...... Feature 25 1 1 0 0 011101 1 2 0 1 0 010011 0 3 1 0 1 101001 1 and I have used the ...
1 vote
1 answer
595 views

How to approach Peak picking with a wide range of peak shapes, sizes, varying noise level, and occasionally shifting baseline?

I am trying write a program that continuously tracks the location a peak. To do that I need a very good peak detection algorithm. It not only has to tell the location of the peak but also the absence ...
2 votes
1 answer
250 views

K-Medoid Clustering with Point Weights

I asked the same question at Cross Validated, here I implemented a K-Medoid clustering algorithm recently; I have a number of points $x_1, ..., x_n$ which have various properties and a distance ...
0 votes
1 answer
156 views

Clustering cartesian coordinates associated with 1 categorical feature

I have a series of 2D coordinates X = {x, y}. Each are associated with one categorical variable W that can take 7 different values. E.g: ...
3 votes
1 answer
331 views

Finding dominating attributes with in the clusters generated

I am having a dataset of customers where each customer is represented as some feature vector and I am applying K-means algorithm to this dataset. On the basis of those features, I can abstract and ...
2 votes
1 answer
579 views

In Orange Data Mining, how do I use results from clustering a training-set to test and score a test-set?

I am performing analysis on the well-known 'Adult' data-set, available on UCI using Orange Data Mining. In a PhD thesis, Pelleg (2004; pg 79) uses unsupervised clustering of the prescribed training ...
0 votes
1 answer
138 views

Stability of clusters in a unsupervised machine learning

I am new to Unsupervised learning. I am working on a customer segmentation data (with no labels). I have done K-Means and also calculated the silhouette score for the model. Now I want to study, if ...
0 votes
0 answers
5 views

How do I determine syndicate and collusion indication with clustering and network analysis on a large unlabeled user transaction data?

How do I determine syndicate and collusion indication with clustering and network analysis on a large unlabeled user transaction data? So far, I've only been training with labeled data on fraud-...
1 vote
1 answer
514 views

Time-series clustering Quality Measures

I am clustering time-series datasets which are not labeled (No Ground truth) and I want to measure the quality of the clusters. Could you please suggest any Clustering performance evaluation methods ...
1 vote
3 answers
123 views

What is the most effective unsupervised ML algorithm to use when outliers are present in data set?

I am analyzing a portfolio of about 225 stocks and have gotten data for each of them based on their "Price/Earnings ratio", "Return on Assets", and "Earnings per share growth". I would like to cluster ...
1 vote
3 answers
154 views

Real-Time Outlier/Anomaly Detection?

My data is the usage/playing statistics for players of a specific game. One data point for a user is aggregated statistics for one week. The goal is: to detect when the player's account was stolen/...
1 vote
3 answers
544 views

I have 32k black and white images. Want to do clustering on them

As the title says I'm trying to do clustering on a set of black and white images. These images are all 200x200 with black dots on a white canvas Example pics here (These are not actual photos from the ...
2 votes
1 answer
487 views

Clustering using both text and numerical features

I have a dataset that contains 2 types of features, one is generated from doc2vec and one is numerical feature. I would like to perform clustering analysis on them. However, due to the size of doc2vec ...
0 votes
1 answer
128 views

Word2vec to encode medical procedures when using isolation forests

I am planning to use Isolation Forests in R (solitude package) to identify outlier medical claims in my data. Each row of my data represents the group of drugs that each provider has administered in ...
0 votes
2 answers
409 views

Topic models for non-textual data?

I am looking to employ an unsupervised clustering on a dataset where each observation has a mix of textual and non-textual features. For each observation, I combine the features into a single vector ...
0 votes
1 answer
65 views

Similarity matching between two distinct datasets (marketing case study)

I am working for a company that sells different products to customers. My objective is to find customers that are likely to purchase product X based on the profiles of customers that already purchased ...
3 votes
2 answers
758 views

Is it possible to make a label automatically in supervised learning(Machine Learning)?

My background knowledge: Basically, supervised learning is based on labeled data. Using the labeled data, the machine can study and determine results for unlabeled data. To do that, for example, if we ...
1 vote
1 answer
120 views

Classifying variable types on a list of variables

I have a list of around 700 variables which I need to perform a variable cleanup on. What complicates things is there are different numeric codes which flag an invalid value and these differ by the ...
0 votes
1 answer
761 views

Restricted Boltzmann Machine (RBM) implementation in Tensorflow (TF) 2.x

I‘m looking for a Python implementation of a Restricted Boltzmann Machine (RBM), e.g. applied to MNIST data as mentioned in „Elements of Statistical Learning“ Ch. 17, in Tensorflow 2.x. I‘m aware of ...
0 votes
0 answers
10 views

unsupervised clustering followed by modeling each cluster to create a mixed model

I am curious if this is an advisable approach. I am not applying this approach and am only interested in the theory of it. let's say you have some set of features X and target Y. X can account for ...
1 vote
1 answer
1k views

Clustering for Sparse Data Matrix of high dimension

I currently have a dataset of 1000 entries with 512 features that are sparse. I want to cluster them. I have attempted using kmeans, but found that the clustering wasn't very good, and have been ...
1 vote
1 answer
634 views

Difference between Q-learning and G-learning in Reinforcement Learning?

What is the difference between Q-learning and G-learning in Reinforcement Learning? Please explain with formulas. An example source: Instead of relying on a utility of consumption, we present G-...
0 votes
1 answer
608 views

Detect time pattern in sequence of events

I have a time series with a timestamp and an associated event: Time Event 1 A 2 B 3 C T A I was wondering if there is a technique/method to figure out which events most often precede others in a ...
0 votes
2 answers
420 views

Time Series - Anomaly Detection

I have time-series data with alerts (every minute) that I need to find anomalies in. I am looking for a library which can do unsupervised learning of this data and detect anomalies in the data. Which ...
2 votes
1 answer
971 views

Dendrogram: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

I am trying to plot a Dendrogram to cluster data but this error is stopping me. My datea is here. I first chose columns to work with: ...
1 vote
1 answer
90 views

Conceptual clustering with sklearn?

How can I perform conceptual clustering in sklearn? My use case is that I have English Wikipedia articles that I'm doing unsupervised learning on (tfidf -> truncated svd -> l2 normalize), and I'd like ...

1
2 3 4 5
10