Questions tagged [clustering]

Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters). It is a main task of exploratory data mining, and a common technique for statistical data analysis, used in many fields, including machine learning, pattern recognition, image analysis, information retrieval etc.

Filter by
Sorted by
Tagged with
3
votes
2answers
434 views

Clustering a very large number of very small clusters with most data unrelated

I'm trying to detect duplicates in a data set of about 34k distinct items. When I say "duplicate," I don't mean identical items, just very similar. I have an algorithm that will Cartesian join the ...
2
votes
1answer
82 views

Find average sequence from a set of sequences [closed]

I have a set of user sessions. Session consists of an ordered list of types of actions that user made (for example, bought a gun, played a mission, etc). I want to create/calculate session that have ...
1
vote
1answer
241 views

What could be a right datascience approach of doing fuzzy string matching in large amount of short text data?

I have two lists to compare. List 1 contains strings which represent the strings in List2. Sometimes, they are direct matches, ...
0
votes
1answer
789 views

Difference between rand index and adjusted rand index?

I am unable to understand what is the adjusted term in ARI. The expected index term in the ARI is from a prior. Kindly explain
0
votes
1answer
58 views

Appropriate Clustering Algorithm

I need to find a good clustering for this data using sci-kit. KNN is not appropriate as it creates blobs although these data are linearly separated. ...
0
votes
1answer
5k views

Implementation of Gaussian Mixture Model for clustering when dealing with multidimensional hyperspectral data in python

I have a python numpy array of size (800,800,4) which is my hyperspectral camera data. When I try standard GMM methods from scikit-learn I get an error saying that the expected dimension of the data ...
2
votes
2answers
4k views

Multidimensional Dynamic Time Warping Implementation in Python - confirm?

I believe that I implemented MDTW in python here but I don't know if I did it correctly. The results seem intuitive. Can someone look at this code and tell me if you see anything wrong? A lot of the ...
1
vote
1answer
571 views

How to compare performance of Cosine Similarity and Manhatten Distance?

I'm doing clustering of documents by applying k-Means on the word-vectors. To measure the cluster quality, I calculate David Bouldin Index for different k's. I tried two different distance measures, ...
1
vote
0answers
63 views

Density Tree - What is the x axis?

I am looking at density trees The intuition about the y-axis is clear: the tree indicates the modes which then merge at merge height: $$m_p(x,y) = \sup{t: \exists C \in \textit{C} \quad s.t. \quad x,...
1
vote
1answer
64 views

Similarity between an n-dimensional curve and selected subsets of them

I'm asking here to get some advice. My goal is to detect similar patterns in an n-dimensional data. For example (extreme simple) you have two axes: X-Axis is Time (in seconds) and Y-Axis is Power (in ...
2
votes
1answer
694 views

What is the Space & Time Complexity of Mini-Batch K-Means clustering algorithm?

For vanilla K-Means clustering algorithm I know that the time complexity is: Time complexity: O(tknm), where n is the number of data points, k is the number of clusters, and t is the number of ...
0
votes
1answer
5k views

clustering multivariate time-series datasets

I am new to clustering.i have data from quality testing of an automobile manufacturing company. I have 100000 datasets.each dataset has 4 variables force, voltage, current, distance. each variable is ...
-1
votes
1answer
42 views

Clustering individuals with random observations

I'm currently trying to apply clustering algorithm to data on callcenter employee KPIs. My dataset contains daily observations per employee on 3 KPIs. Yet the challenge is that I have a random number ...
2
votes
1answer
5k views

How to interpret agglomerative coefficient agnes() function?

how to interpret agglomerative coefficient in agnes() function of cluster package? Example: ...
5
votes
2answers
3k views

Clustering or classifing n-gram-based text categories

I have large set of data records looking like this: "text", "category" I extract n-grams from text (2-, 3- and 4-grams) and store count of each n-gram per ...
0
votes
2answers
363 views

Scalar entities for k means clustering

I am trying to understand kmeans clustering and I read a article where kmeans is used for clustering the features generated in network logs. This clustering is followed by a supervised classification. ...
1
vote
2answers
5k views

Difference between Global Outlier and Contextual Outlier?

I am studying "Data Mining: Concepts and Techniques" by Han, Kamber & Pei. In Chapter 12 "Outlier Detection", they have stated that there are 3 types of outliers: Global Outlier - deviates ...
0
votes
2answers
296 views

How are the positions of the output nodes determined in the Kohonen - Self Organizing Maps algorithm?

In the Cooperative stage of Kohonen's SOM, the neighborhood for a winning neuron(output node). In most cases, the neighborhood function happens to be the Gaussian Function. For example, $$h_j,_i = exp(...
0
votes
1answer
217 views

Clustering Multiple Networks

I'm looking for methods of Community Detection in networks. For example if I have a network of 100 people (each node is a person), how can I cluster nodes? What would be the best approach for grouping ...
-2
votes
2answers
285 views

How is PCA is different from SubSpace clustering and how do we extract variables responsible for the first PCA component?

New update: I understand PCA components ensure we select variables responsible for high variance, but I would like to know how to extract key variables responsible only for high variance through PCA ...
-1
votes
2answers
367 views

Which Clustering algorithm to use for unique 4Dimension dataset before feeding to correlation?

Lets give an example X: 1 2 3 4 5 Y: .9 .91 .92 .93 .94 Z: 20 36 999 211 M. 4000 3456 1 0 When I have such dataset, Which clustering algorithm to choose ? Also, ...
1
vote
2answers
275 views

Dynamic clustering for text documents

I have few hundred thousands of text documents. Some of them are pretty similar - they differ just in ex. names or some numbers, all other text is the same. I would like to cluster these documents, so ...
0
votes
1answer
206 views

How to Interpret the output of PCA?

I have dataset of 50000 values (rows) and 1000 variables (columns). Since this is high dimensional, I am unable to work with just DBSCAN. So I am trying to use PCA (principle component analysis). ...
1
vote
4answers
10k views

Categorical data in Kmeans [duplicate]

I need to perform clustering in a given dataset. There are atrributes with numerical as well as categorical values. What is the best way to convert categorical to numeric value ? as an example one ...
1
vote
0answers
1k views

SOM initial values for learning rate and neighborhood sigma

I am using SOM (Self-Organizing Maps) of Kohonen, or more specifically, the MiniSom, found here to cluster and visualize my data. As you can see in the above site, the example given is: ...
1
vote
3answers
182 views

Alternative methods for improved clustering separation?

I have the following labeled cluster, which is what an ideal clustering algorithm would generate: Now, I have applied a basic K-Means clustering algorithm to the data, and the outcome is as follows: ...
6
votes
3answers
26k views

How to test accuracy of an unsupervised clustering model output?

I am trying to test how well my unsupervised K-Means clustering properly clusters my data. I have an unsupervised K-Means clustering model output (as shown in the first photo below) and then I ...
3
votes
2answers
4k views

Outlier detection by unsupervised algorithm: Fraud Detection

I have set of 300,000 set of rows with credit card transactions and my job is to find outliers (suspicious transactions) in those dataset. I have created around 5 features (All continuous data, with ...
7
votes
3answers
7k views

Why do we use a Gaussian kernel as a similarity metric?

In graph-based clustering, why is it preferred to use the Gaussian kernel rather than the distance between two points as the similarity metric?
4
votes
2answers
2k views

How relevant is Self Organizing Maps in today's science? [closed]

Self-Organizing Maps is a pretty smart yet fast & simple method to cluster data. But Self-Organizing maps were developed in 1990 and a lot of robust and powerful clustering method using ...
3
votes
1answer
60 views

How to compare the performance of different number of mixing components for EM algorithm?

I am reading about the EM (Expectation-Maximization) algorithm in a machine learning book. At the end remark of the chapter, the authors mentioned that we cannot decide the "optimality" of the number ...
3
votes
2answers
2k views

Efficient clustering of sparse binary vectors

I'm trying to perform clustering of the data to improve the efficiency of brute-force kNN . The dataset consists of objects described by a large set of binary features, each identified by a 32-bit ...
2
votes
3answers
7k views

PCA before K-mean clustering

If I applied PCA on feature vectors and then I do clustering, such like following: ...
1
vote
0answers
175 views

Seeking Appropriate Clustering Algorithm

I'm analyzing the GDELT dataset and I want to determine thematic clusters. Simplifying considerably, GDELT parses news articles and extracts events. As part of that, it recognizes, let's say, 250 "...
1
vote
0answers
50 views

Looking for an algo transforming numerical attributes into categorical attributes -cleverly

I created an algorithm which works on categorical attributes. The input data comes with categorical attributes, but numerical ones as well. How can I apply a pre-processing which transforms the ...
3
votes
4answers
265 views

Distance measure for ternary feature

I have a data set consisting of 100 features each of which are ternary: values of -1 if it exists in one category, 0 if it doesn't exist, and 1 if it exists in the second category. For example ...
4
votes
0answers
92 views

Fixed-radius range search in non-Euclidean space

I'm trying to find an indexing data structure most suitable for my metric space: set of IP network related data (IP addresses, ports, TCP flags, ...), distance function is continuous, non-Euclidean ...
3
votes
1answer
87 views

Weighted degree in Multidimensional networks

Does there exist a definition for weighted degrees of multidimensional networks? I understand that the basic definition would be: Let $v\in V$ be a node of a network $G=(N,V,L)$. The function ...
1
vote
3answers
141 views

Memory-efficient metric calculation for ultra high dimensional data

I am preparing for clustering the data which can be only represent as a extremely sparse binary vectors. Each of the objects is represented by a large set the binary features ($10^3$ ~ $10^6$), each ...
2
votes
1answer
1k views

HDBSCAN cluster: still unclear to me how to chose 'min_cluster_size`

Hdbscan is an excellent technique to find the "optimal" number of clusters within your data when you have little a priori idea how many clusters should exist. This makes the method great for ...
8
votes
1answer
55k views

Confused about how to apply KMeans on my a dataset with features extracted

I am trying to apply a basic use of the scikitlearn KMeans Clustering package, to create different clusters that I could use to identify a certain activity. For example, in my dataset below, I have ...
3
votes
3answers
267 views

Is Clustering used in real world systems/products involving large amounts of data? How are the nuances taken care of?

I am working with a real dataset of around 2.5 million short text data (posts, around 10 words in an average data element). I wanted to group similar posts together. So, I applied KMeans clustering on ...
4
votes
1answer
863 views

Given a t-SNE plot, how can I infer the “most correct” labels? How does one understand its structure?

Let's say I begin with an exceptionally large dataframe (e.g. imported/munged from tsv files). Several of these columns are categorical labels. (As a more concrete example, let's imagine a group of ...
8
votes
2answers
9k views

Clustering high dimensional data

TL;DR: Given a big image dataset (around 36 GiB of raw pixels) of unlabeled data, how can I cluster the images (based on the pixel values) without knowing the number of clusters ...
3
votes
1answer
132 views

What kind of classification should I use?

I'm very new to machine learning (and statistics) and I'm struggling with some basics (I'm using R as my primary environment). I have a lexicon with words and all of their forms (in different ...
-1
votes
1answer
1k views

KMeans clustering to help label Multi-class Supervised model

EDITED: Is it accepted practice to be able to use a KMeans clustering algorithm to help label data fed into a supervised model? (Unsupervised --feeds-> supervised)? The reason being, relabeling ...
2
votes
2answers
180 views

Using a K-NN Classification Approach for Time Series Data?

I have a dataset which contains time-series data of water flow over time. I have a flow meter connected to a kitchen faucet, and I am trying to cluster or classify specific water usage events. The ...
2
votes
0answers
621 views

Mixed geospatial and categorical clustering

I'm working on a project that seeks to identify clusters in urban development based on location (in lat/lon) and a categorical variable (what the particular site is zoned for). Ideally, the analysis ...
3
votes
1answer
716 views

Predict a tree structure out of nodes with different features

The Problem Suppose we have a representation of a document's text layout as in the image below: Here, each rectangle represents a chunk of text (one word, an expression or even part of a sentence). ...
0
votes
3answers
616 views

Deciding the number of clusters in K-means clustering of descriptors

I am new to the Machine Learning area and I have a question to ask. But let me first post the problem. Problem: The problem is very simple. I want to classify images as Category-1("Images containing ...

1
13 14
15
16 17
20