Questions tagged [clustering]

Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters). It is a main task of exploratory data mining, and a common technique for statistical data analysis, used in many fields, including machine learning, pattern recognition, image analysis, information retrieval etc.

Filter by
Sorted by
Tagged with
0
votes
1answer
42 views

Any cluster algo can cluster time series datasets based on variation ratio(or quantity)?

I learn machine learning from sciki and read its documents. Clustering cluster groups based on the euclidean distance and filter them by different ways ex: guassian distribution, or mean-shift...etc. ...
17
votes
1answer
12k views

Algorithms for text clustering

I have a problem of clustering huge amount of sentences into groups by their meanings. This is similar to a problem when you have lots of sentences and want to group them by their meanings. What ...
1
vote
2answers
28 views

Clusterize item set with items as vectors of features

I have to clusterize this dataset in which I have houses and water consumption in this form: $$ House1 = (x_{1},x_{2}... x_{n});\\ House2 = (y_{1},y_{2}... y_{n});\\ House3 = (z_{1},z_{2}... z_{n});\\ ...
1
vote
0answers
11 views

Normalization before PCA in NLP domains?

I'm working on a basic bag-of-words toy NLP pipeline for sentiment analysis using scikit-learn. From research of other questions here, it seems that the main applicable scaler for before PCA is the ...
3
votes
1answer
56 views

How to do clustering assuring more than one class per cluster?

I have a dataset with 4 classes and i'm trying to use an ensemble model where each base classifier trains with a portion of data. To distribute data along the classifiers, i am using KMeans algorithm. ...
1
vote
1answer
43 views

How to compare topics generated from topic modeling from different datasets?

I have two datasets of a similar theme. Let's assume Dataset A and Dataset B. Using the top2vec model (https://github.com/ddangelov/Top2Vec) (https://arxiv.org/abs/2008.09470) on each dataset, I came ...
1
vote
0answers
11 views

Most useful clustering algorithm for NER / document matrix

I have a matrix composed of documents in columns and named entities recognized in all the documents as rows. K-means clustering has not offer me a meaningful set of clusters, and indeed one of the ...
1
vote
1answer
8 views

Why checking the distribution of data is needed before calculating Gower distance?

I read this article(Clustering datasets having both numerical and categorical variables) to learn how to perform clustering on datasets with not just numerical variables. Before calculating the Gower ...
0
votes
0answers
8 views

Improve autoencoder clustering

I am trying to use an autoencoder to perform dimensionality reduction on a dataset before clustering. The data consists of 5000 samples with 2000 features each. I'm using keras (tensorflow backend). ...
0
votes
1answer
73 views

Can distortion be derived from inertia rather than recalculating it from scratch in case of kmeans?

I got this definitional difference between distortion and inertia from here: Two values are of importance here — distortion and inertia. Distortion is the average of the euclidean squared distance ...
1
vote
1answer
23 views

Clustering sets based on common elements

I am looking for a clustering algorithm for the purposes of combining routing information. Suppose we have the following sets: A={1,2,3,4,5} B={2,3,4,6} C={3,4,7,8} D={8,9,10,11} If we want 3 groups, ...
20
votes
4answers
18k views

When to use cosine simlarity over Euclidean similarity

In NLP, people tend to use cosine similarity to measure document/text distances. I want to hear what do people think of the following two scenarios, which to pick, cosine similarity or Euclidean? ...
0
votes
0answers
13 views

What type of clustering algorithm to use to determine what accounts belong to a family?

I have both categorical (Name, address, etc.) and numerical data (similarity scores between two parameters) and I can't figure out what kind of clustering ML algorithm would be appropriate since most ...
1
vote
1answer
18 views

Clustering on binary data

I am working on clustering on binary data which has 25 features, sample Feature 1 Feature 2 Feature 3 ...... Feature 25 1 1 0 0 011101 1 2 0 1 0 010011 0 3 1 0 1 101001 1 and I have used the ...
0
votes
1answer
70 views

clustering people according to answers on survey

Hi I am finding it hard to find online the best clustering algorithm for clustering people according to answers they gave on 20 question survey. There are four categories which each of these answers ...
0
votes
1answer
20 views

Distance between any two points after DBSCAN

DBSCAN is a clustering model which is robust to detect the outliers also. A parameter $\epsilon$ i.e. radius is an input of the algorithm, a point is said to be outlier if it's circle with radius $\...
1
vote
1answer
62 views

How to use spectral clustering to predict?

In an academic paper, they talk about using a nearest neighbour algorithm to predict the cluster of a new point. And how the number of nearest neighbours is set to 10 in their example. What do they ...
2
votes
1answer
31 views

How to retrain a K-Modes model based on daily data?

I have read that retraining a model depends highly on what you are trying to achieve. I am conscious that maybe I need to retrain my model daily and after a certain time I have to train the model ...
0
votes
1answer
26 views

K-means clustering with categorical data

I am doing a clustering analysis using K-means and I have around 6 categorical variables that I want to consider in the model. When I transform these variables as dummy variables (binary values 1 - 0) ...
0
votes
1answer
167 views

How can i use Hellinger Distance on array of different length?

I have to use Hellinger distance to compare arrays that are not the same length. How do you do this correctly? Putting a zero in the missing fields for the shorter array does not sound like the best ...
0
votes
1answer
17 views

Real-Time Outlier/Anomaly Detection?

My data is the usage/playing statistics for players of a specific game. One data point for a user is aggregated statistics for one week. The goal is to be able to detect when the account of the player ...
1
vote
1answer
42 views

How to properly train your Self-Organized Map?

I recently stumbled upon the Self-Organized Map - an ANN architecture used to cluster high dimensional data - while simultaneously imposing a neighborhood structure on it. It is trained through a ...
0
votes
1answer
45 views

How to increase number of outliers in a dataset?

I have a dataset with 1000 rows and 4 columns with 3 outliers .I want to add another 7 outliers related to them for detection by clustering. ...
2
votes
1answer
32 views

How can I reduce the number of dimensions using a Clustering algorithm in a mixed dataset?

I am working with a mixed data set, corresponding to TV consumption data, with the aim of reducing the number of features to only those relevant to detect TV consumption patterns (or consumption ...
1
vote
1answer
154 views

How to structure my data into features and targets for PCA on Big Data?

I want to apply the PCA algorithm from Scikit-Learn.(https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html ) At the part where I have to separate the features and the ...
0
votes
1answer
14 views

Does scaling required for this kind of datasets?

I have a dataset with features like views of a product (in hundreds of thousands), clicks on the products (in thousands), conversion rate (in decimal such as 7.6%) and sales (in hundreds). Do I need ...
0
votes
1answer
41 views

Speaker clustering/diarization

I am working on the problem of speaker clustering. I am using k-means clustering. The ground truth cluster values and k-mean cluster values do not correspond due to different methods of labelling (...
0
votes
1answer
19 views

What kind of algorithms can be run on this kind of dataset?

We are running a survey majorly on our ordering system like delivery, curbside and in-store pickup. We are collecting the ratings for ordering experience, how likely the customer would recommend our ...
1
vote
2answers
390 views

Intuition behind the fact that SVM uses only measure of similarity between examples for classification

I have read about SVM and although I did not understand the math behind it completly, I know that it produces decision plane with maximum margin between examples of different classes and role of ...
0
votes
0answers
22 views

Best approach to clustering images

I am new to unsupervised clustering and I wish to perform clustering on a dataset of 512 images. I want to output n clusters where each cluster holds images that are similar to each other. I do not ...
0
votes
2answers
507 views

How to cluster some text using TensorFlow

Apologies for any inaccuracies due to the infancy in this field. I'm trying to learn on how to return a dataset with three classes to be clustered by TensorFlow. At this stage, I've read a lot and ...
3
votes
2answers
1k views

K-modes implementation in pyspark

I'm looking for an implementation of k-modes in pyspark. I found this and this as implementations. First, I tried implementing k-modes using the first link and faced issues. So I went ahead and tried ...
0
votes
0answers
6 views

How to check if 2 images where splitted?

There is an original data-set of images. Each image was splitted into 2 parts (left-right). I want to run on all those splitted images and check if each 2 images are spliited from same image. Is ...
2
votes
1answer
42 views

Clustering Weekday Weekend Data and Multicollinearity

Hi I have data of weekday and weekend step counts in which I extracted metrics from them such as the wd steps, we steps, standard deviation of wd steps, standard deviation of we steps and so on... <...
2
votes
1answer
424 views

Dendrogram: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

I am trying to plot a Dendrogram to cluster data but this error is stopping me. My datea is here. I first chose columns to work with: ...
1
vote
1answer
30 views

Determining number of clusters in high dimensions

I am doing KMeans clustering for sentence embeddings and my problem is the number of clusters. In general, feature size is an order of a few hundreds (in this case 768) and my concern is the sparsity ...
2
votes
2answers
153 views

A suitable feature vector for images

I have a set of images of various products from different websites. I want to cluster the images based on the product shown in the image. How can I generate a suitable feature vector for an image for ...
1
vote
0answers
17 views

Unsupervised Learning with audio recordings

I had a (probably crazy) idea for a project and I was wondering if you all think it would be in any way possible. I'm interested in analyzing sounds made by different types of animals (for example ...
0
votes
1answer
22 views

In clustering, sequence number such as customer ID and dates such as purchase date should be dropped?

I am learning K-means clustering and found that in most datasets, there are sequence number such as customer ID and dates such as purchase date. I don't see any use in them for clustering. Should I ...
1
vote
1answer
50 views

What is the best way to cluster this kind of data?

I have data that looks like this: The chart on the left is the trend and the smaller chart on the right is the box plot showing the distribution of means. Each color is the output of a particular ...
1
vote
1answer
73 views

Visualising K-Means clusters for 3D data in R

I have an excel file that contains 485k rows x 3 columns of integer values. Sample data: ...
2
votes
2answers
46 views

Identifying common keyphrase frequency in large dataset

I have a dataset of profiles that contain freeform text describing the work history of a number of individuals. I would like to attempt to identify frequently used words or groups of words across the ...
2
votes
1answer
203 views

How to get the probability/closeness of a sample belonging to a specific cluster?

I'm new to this so please let me know if my logic of comparing cosine similarity and k-means is incorrect I got a set of ...
0
votes
0answers
20 views

what is the best distance measure to use for clustering high dimensional binary data?

I have a dataset where the input is a dataset for ICU patients where each ICU stay has 40 features (20 vitals, 20 numerical lab values) and multiple time steps (the stays' length is between 6 and 19-...
0
votes
1answer
36 views

Inference from text data without label or Target

I have a use case where I have text data entered by an approver while approving of some loan. I have to make some inferences as to what could be the reasons for approval using NLP. How should I go ...
2
votes
1answer
21 views

Comparing Clustering Over Time

I've recently conducted a k-prototypes R routine on some mixed data. In particular, the data is health data concerning a certain public health intervention, with categorical variables for health ...
0
votes
1answer
96 views

External cluster evaluation for a varying number of cluster

There are many external clustering indices like (Adjusted) mutual information, (Adjusted) Rand index, and many more. However, they are not very good at comparing clusterings where the number of ...
35
votes
6answers
64k views

Calculating KL Divergence in Python

I am rather new to this and can't say I have a complete understanding of the theoretical concepts behind this. I am trying to calculate the KL Divergence between several lists of points in Python. I ...
1
vote
1answer
619 views

score or cost function for AgglomerativeClustering

I am learning AgglomerativeClustering using sklearn. It is fairly easy to use for example: ...
1
vote
1answer
37 views

Customer Segmentation: Should I use a variable, representing a product, that is unpopular in the dataset for K-Means Clustering?

I am working with a data set that, besides customer age and income, tells the balance a customer has in different type of bank accounts: Checking, Shares, Investment, Savings, Deposit, Mortgage, Loan, ...

1
2 3 4 5
24