Questions tagged [clustering]
Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters). It is a main task of exploratory data mining, and a common technique for statistical data analysis, used in many fields, including machine learning, pattern recognition, image analysis, information retrieval etc.
1,369
questions
0
votes
0
answers
11
views
Do you have to use clustering with SciKit-Learn's Mutual Information metric?
I'd like to calculate the mutual information between two datasets, but I'd prefer not to cluster them first.
I'm thinking of using SciKit-Learn's mutual_info_score ...
- 511
0
votes
1
answer
15
views
How to cluster based on sensor data? - My first data science job
I'm on my first (real), data, programming job. As everyone can imagine, this can be quite hard and I learn a lot from it, given I am a data science student in university. However, I am completely ...
- 1
0
votes
0
answers
21
views
SequentialFeatureSelector and scoring for clustering
I want to use SequentialFeatureSelector from scikit-learn to do feature selection for clustering (K-Means). [Here][1] is a list of available evaluation scores for clustering. What I do not understand ...
- 1
0
votes
0
answers
10
views
Product scrape data clustering model
I am developing a tool that scrapes multiple websites for products. Some websites sell the products themselves, while others feature products sold by third-party sellers that may have incorrect ...
- 101
0
votes
1
answer
21
views
Statistical test for comparing number of clusters in data
I am performing $K$-means clustering on a dataset consisting of $n$ observations and $d$ variables, and I'm trying to determine the optimal number of clusters. Is there a test that can determine the ...
- 121
0
votes
0
answers
6
views
getting wrong cosine similarity when using face_recognition.faceencodings
I am trying to calculate cosine similarity between two face encodings returned by face_Recognition.face encoding() which 128d vector which always is above 0.8
also is the face encoding normalized ...
1
vote
3
answers
54
views
What's the fastest clustering package in Python?
I'd like to perform clustering analysis on a dataset with 1,300 columns and 500,000 rows.
I've seen that clustering algorithms are available in SciKit-Learn. But I'm worried that the algorithms will ...
- 511
0
votes
1
answer
32
views
Patterns extraction in time serie with DTW
I have a long time serie, let's say 1000 items. I want to find patterns in it of different lengths from 10 to 100 elements.
To do this, I extract sliding windows of different lengths and calculate ...
0
votes
1
answer
9
views
MEL VS linear spectrograms for bioacoustics machine learning
I don't have background in bioacoustics but working on a data-science project in bioacoustics.
I am working with animal vocalizations recorded at sampling rate of 250000.
Animals are bats, which are ...
- 165
0
votes
0
answers
19
views
Assign clusters to new data using a model based on previously gained data
In an experiment, we gained data through an online survey. We used this data to conduct a cluster analysis (using the library(mclust) library).
According to the ...
- 1
0
votes
1
answer
16
views
Scoring vectors by "distinctness"
I have a set of vectors in a vector space and I'm looking for some score that measures how "distinct" each vector is from the others.
I suppose this is a well-known problem but I have no ...
- 101
0
votes
0
answers
11
views
Clustering methods for text and image features
I want to build a recommender system with unlabeled data and used TF-IDF to extract text features from a given short description and VGG-16 to extract image features. I am looking for a way to combine ...
- 1
1
vote
1
answer
25
views
best algorithms for clustering customers, customer segmentation
I have a dataset mixture of categorical and numerical variable, I was wonder what are the best algorithms to cluster customers?
how to find the underlying patterns that segments a customer??
- 13
0
votes
1
answer
29
views
Efficient ways of clustering for big data
I have a task which is customer segmentation with 120k users and a record of their purchases which is +3 million records of data, the approach I want to use is to use clustering algorithms like kmeans ...
- 13
1
vote
1
answer
21
views
Movement in cohorts
I am working on a user sales data which gets updated week over week. Based on the sales done in each week, the user is categorized in segment A, B or C. This means size of each segment could change ...
- 31
0
votes
0
answers
19
views
Cluster-Based Anomaly Detection (and PCA)
I have a dataset of user operations (250 types of operations) in a trading platform and my task is to extract features, flagging rules, other insight for fraud prevention/anomaly detection, or some ...
0
votes
1
answer
14
views
Maximum categories for categorical variables in K-Means clustering
I am trying to perform K-means clustering on a dataset, and one of my categorical features has 96 possible options. Would this be too many features for one variable to have? The alternative would be ...
- 101
0
votes
1
answer
12
views
How can I show data clustering after building a Convolutional Variational Autoencoder?
I am brand new to Machine Learning, and just followed Tensorflow Convolutional Variational Autoencoder Tutorial using my own dataset instead of the MNIST set. I was able to successfully display the 2D ...
0
votes
0
answers
7
views
Analysis to identify similar occupations by frequency of skills requested in job postings (in R)
I have access to a dataset of job postings, which for each posting has a unique posting ID, the job posting occupation, and a row for each skill requested in each job posting.
The dataset looks a bit ...
- 1
0
votes
1
answer
45
views
How to compare labels from clustering analysis and original ones?
I was asked to run a clustering analysis to assess the validity of labels for a manually labelled dataset.
I can simply save the actual labels (4 classes: 0, 1, 2, 3) and run clustering analysis (let'...
0
votes
0
answers
14
views
Select Topic Words from Clusters
I am following this solution for clustering: https://towardsdatascience.com/clustering-contextual-embeddings-for-topic-model-1fb15c45b1bd
For step four "Select Topic Words from Clusters", I ...
- 129
1
vote
1
answer
26
views
Customer Segmentation with mixed data [closed]
I want to perform clustering. I am reading about this topic but I am totally confused. My dataset has 490 observations and it consists of numerical data (3 columns: Recency, Frequency, Monetary), ...
- 11
0
votes
2
answers
29
views
NLP topic clustering
In my dataset, I have 500 abstracts. The goal is to cluster them in 2 topics.
One topic must have those abstracts which contain some list of words or similar words and the rest of the abstracts in ...
0
votes
0
answers
24
views
How assign a new sentence embedding to a cluster in fast clustering
I am working on clustering sentence embeddings obtained from sentence transformer, for which I used fast clustering (clustered on cosine similarity).
https://www.sbert.net/examples/applications/...
- 13
0
votes
0
answers
17
views
Determine unusual occurrence of words in classes
I am working on a project where I have 20+ classes/groups. Each of these groups perform certain text searches. I am looking for specific keywords example 'code' which is an anomaly. The challenge is ...
- 11
1
vote
1
answer
19
views
Which pre-trained model to select to generate embeddings from shop names written in English?
Good afternoon!
I have a dataset with thousands of shop names written in English. Several shop names might belong to one business entity, for instance, shops with names "KFC 001", "WWW....
- 11
0
votes
1
answer
57
views
Cluster products that are frequently bought together
I have a dataset of articles metadata for each article, so something like this:
product_id
color
type
1234
red
t-shirt
and another containing the transactions of customers, which looks like this:
...
- 1
0
votes
1
answer
39
views
In DBSCAN, can the distance between a Noise Point and Border Point be less than Epsilon?
In DBSCAN:
A core point is a point which has at least "MinPts" points inside its Epsilon radius.
A border point is a point inside the Epsilon radius of a core point, but it has a number of ...
0
votes
0
answers
21
views
Ways to measure similarity between two datasets with different clusters
I have two datasets, each one with a different set of clusters of different shapes and sizes. What are some ways to compute the similarity of the clusters of these two datasets? (Note that both ...
1
vote
0
answers
13
views
How to do unsupervised clustering on sentences to find intents
I am working on chatbot for students.
So, I have chatlogs on conversation between student and tutor, which is on mathematical problems (no labels).
First thing I want to find is intents, such as ...
- 13
2
votes
1
answer
34
views
Identifying competition climbing styles from World Cup data
In competitive climbing, it is well known that there are many different styles, differentiated by e.g. the steepness of the climbing wall. Some athletes excel at certain styles more than others. I ...
- 121
0
votes
0
answers
14
views
Finding centroids with less difference but high variability
I have a data set including lat , lon and id's for some places. Also I have another table which has features for each id's.
I wanted to find cluster centroids for those places. Using lat and lot ...
- 53
0
votes
1
answer
38
views
Cluster URLs based on their pattern
I am new to clustering techniques and I highly value any input you can provide for my problem bellow.
Basically, I want to cluster URLs based on their structural patterns.
for example
cluster1 - ...
- 101
0
votes
1
answer
26
views
Clustering on Raw Image Pixel Array
I have an array in the shape of (105, 105, 3). When I do plt.imshow(array) it outputs:
How can I run a clustering algorithm directly on this image? Do I need to convert the pixels to cartesian ...
0
votes
0
answers
12
views
How to quickly know if there is a cluster or several clusters in a certain area?
In my research experiment, I want to use genetic algorithm to optimize a set of 2d points in an area. The objective is to make those points to be less average scattered. It means that there should one ...
0
votes
1
answer
19
views
How to isolate a clear relationship from a subset of data with lots of noise and outliers
I am doing an analysis of aircraft data and I want to see how much fuel is burnt on landing. There are 2 main factors aircraft type and landing time (ie. time elapsed)
However there is a cheeky third ...
- 23
0
votes
1
answer
32
views
What can be the reasons for 95% of samples belong to one cluster when there is 5 clusters?
'''I used the k-means algorithm to clustering set of documents which are textual data only.
The document has 2lack records.
Surprisingly the result for the clustering is
90% of records is storing in 1 ...
- 1
0
votes
0
answers
10
views
What does near-cut (threshold) graph say about original complete weighted graph?
CROSSPOST: https://math.stackexchange.com/q/4588249/117548
ORIGINAL post, preamble
Start from a complete graph with weighted edges (e.g. in $[0,1]$ interval). Continuously increasing threshold $t$ ...
- 21
0
votes
0
answers
50
views
How to do DBSCAN clustering with mixed variables (numerical features and binary/ordinal variables)?
I have a question written at the end of the post which refers to the "Distances" paragraph. The other first two paragraphs give additional info.
Context
I'm working on a project where I have ...
5
votes
2
answers
415
views
Dynamic clustering in NLP dataset
Let's imagine this list:
corpus = ['cat','banana','dog','horse','apple','tiger','snake']
I am looking for a way to build a vectorizer with dynamic clustering. I'm ...
- 51
0
votes
0
answers
11
views
How to graphic evolution of clusters in time
I am currently working in the definition of the profiles of patients of a specific disease. I have applied an unsupervised clustering technique to my dataset obtaining different patient profiles for ...
2
votes
1
answer
77
views
Incremental clustering algorithm
I am looking for an incremental clustering algorithm.
By incremental I mean an algorithm that builds clusters starting from an initial dataset and that is able to progressively ingest new items/...
- 121
0
votes
0
answers
17
views
Clustering mixed type variables with Orange
I wonder if with Orange it is possible to cluster mixed type data, so a dataset with numeric as well as discrete (categorical) data (ordered / unordered). Can you show an example of how that could be ...
0
votes
1
answer
15
views
How to identify new potential customers from existing customers profile?
I have firmographic data of all the possible customers. Data includes sales, profitability, capital, organisation size, geographic location, industry, etc.
What is the best way of identifying new ...
- 233
0
votes
0
answers
49
views
Dendrogram in hierarchical clustering
I have a confusion regarding the y-axis of the dendrogram. In many references, I see that the y-axis of the dendrogram is labeled as the Euclidean distance like the one shown below :
But according to ...
- 35
0
votes
0
answers
5
views
How to check how similar or different two texts are based on their cluster membership?
I have generated clusters based on crowdsourced ratings of a set of comments (collected from two different platforms). Each comment was rated by three different annotators using a likert scale of 3 ...
- 1
0
votes
1
answer
16
views
Find clusters of industrial probe
I'm new to data science, but in these last months i have started to gather many samples of values coming from different industrial probes (temperature of water, pression, Kw consumed, etc). I have ...
0
votes
0
answers
30
views
Using mean absolute SHAP values as input into UMAP
Imagine you have a data set and you calculate SHAP values to understand feature importance. You then calculate the mean absolute SHAP values, as per this bar chart in the Interpretable ML book.
If you ...
0
votes
0
answers
11
views
Why is exactly that k-mediods or PAM works well in a non-metric space
While brushing up on k-means and related topics I read about the K-mediod/PAM clustering algorithm. One of the key points why it was an interesting algorithm, compared to k-means was that it worked ...
0
votes
0
answers
21
views
How to filter noise from piecewise linear dataset
Assume that a vehicle is moving with a constant speed, and the observed path is given by a set of points
$$(X, Y) = (x_i, y_i)_{i=1}^{N} \in \mathbb{R}^2$$
We know that:
observations are noisy: if $(...
- 101