New answers tagged

0

By googling: HDBSCAN is order of n squared whereas optics is order of n times log(n).


0

You are describing incremental learning, input data is continuously used to extend the existing model's knowledge. There is a Python implementation of incremental DBSCAN. There is no current Python implementation of incremental HDBSCAN.


0

I suggest you to use an Autoencoder for dimensionality reduction. An Autoencoder is a Neural Network with a hourglass shape, that is meant to learn a compressed representation of your data. You can train it first on the data you already have, and then use it to extract a compressed representation at a time. In your case, what you need is an Autoencoder with ...


0

It sounds to me like what you should really do is train a (multi-class) classifier on the dataset, and then use it to 'predict' each new incoming face. If you don't have another source of labels, you can use your DBScan result as a label (i.e. use the cluster as a class label). That being said, you technically can check a new data sample by comparing in to ...


1

So the question is asking why the first two principal components of your encoded text data is encapsulating all of the variation in the data. One potential issue could be the averaging over word vectors. Suppose for a particular feature across word vectors for a particular post f, there could be an array of positive and negative values. When we then apply an ...


0

You can use distance metrics that are better suited to boolean values such as Jaccard, or Manhattan. Usually K-means uses mean as the metric to re-adjust the centroid of the current cluster, but I think you could change this to majority vote. I think if you one-hot encoded the categorical features, you could use Jaccard for similarity and then majority vote ...


0

For the tags: Do you know how they are generated? How many unique tags do you have? If they are self-generated (ie lots of tags that can be subsets of other tags). You might need to do tag consolidation which will also help with dimensionality reduction of the word vectors. If you can provide a bit more information about what your data looks like and where ...


1

My take after a quick read of the references. First of all, Lance and Williams's original paper mentions that their linear scheme works (and offers computational advantage) only for combinatorial strategies. Is minimax linkage such a combinatorial strategy? In other words, does it depend (linearly) on pair-wise distances? By the defintion of minimax distance ...


0

I think that your question is how to find the attributes of a model (parameters are the ones used to tune the model). You can find the Model attributes from the Scikit-learn documentation of that model in the Attributes section. Attributes for K-Means: cluster_centers_: ndarray of shape (n_clusters, n_features) Coordinates of cluster centres. If the ...


1

I believe you are trying to access "labels_" before fitting the data. from sklearn.cluster import KMeans import numpy as np X = np.array([[1, 2], [1, 4], [1, 0], [10, 2], [10, 4], [10, 0]]) kmeans = KMeans(n_clusters=2, random_state=0).fit(X) def get_properies(model): return [i for i in model.__dict__ ] get_properies(kmeans) ...


0

you can find the list of an algorithm's attributes from the sklearn page for the corresponding algorithm. For Kmeans: Attributes cluster_centers_ :ndarray of shape (n_clusters, n_features) Coordinates of cluster centers. If the algorithm stops before fully converging (see tol and max_iter), these will not be consistent with labels_. labels_ :ndarray of ...


0

you can try TSNE which is a common visualisation form to understand high-dimensional data in 2D and 3D. TSNE is a better starting place to start with for 3D visualization.


2

My answer would be second option I think the use of PCA is to represent original high dimensional information/data in lower dimension by calculationg the direction/axes along which there is maximum variablity in data. In first case, where you filter for 0-labeled observations and then do PCA so PCA would measure variablity based on a smaller version of ...


1

max_iterint, default=100 Maximum number of iterations over the complete dataset before stopping independently of any early stopping criterion heuristics. It's the number of iteration over the full dataset. Number of partial_fit will depend on batch_size batch_sizeint, default=100 Size of the mini batches. partial_fit(self, X[, y, sample_weight]) Update k ...


0

I think it would make sense to compare the distribution (and means) of standard deviations for each variable between the groups. So the mean of the standard deviation for each era would be something like: $ \overline{\sigma}_{Era} = \frac{\sum_{i = (features)} \sigma_{i}}{n} $ where $n$ is the number of features for each era, and $ \sigma_{i} $ is the ...


0

A similar post appears on Cross-Validated, "Estimating the most important features in a k-means cluster partition". Quoting from that post: One way to quantify the usefulness of each feature (= variable = dimension), from the book Burns, Robert P., and Richard Burns. Business research methods and statistics using SPSS. Sage, 2008. (mirror), ...


0

The numpy calculation is the correct one to use, but may be a bit tricky to understand how it is calculated Your custom calculation is accidentally returning the inverse slope, the x and y values are reversed in the slope function (x1 -> y[i], etc). The slope should be delta_y/delta_x def slope(x1, y1, x2, y2): v=slope(y[i], x[i], y[i-1], x[i-1]) Also, ...


0

I was thinking about this very question this week, and I had an idea. Forgive me if this is way off. Suppose an n dimensional dataset to be sorted into k clusters. If we have a 3 layer network: input layer: n neurons hidden layer: k neurons, softmax activation output layer: n neurons, linear activation If we use our dataset as both x and y (this is a typical ...


0

Not all the parameters (e.g., the assignment parameters) for a Gaussian mixture model are smoothly differentiable, thus can not be fit with gradient descent. Other use cases for the expectation–maximization (EM) algorithm are: Clustering Latent variable estimation Missing data estimation


1

That is a data mining problem, specifically affinity analysis. One common method to solve it is the Apriori algorithm.


1

Theoretically, Gaussian Mixture Model could identify [\, /] clusters.


2

It appears to me that what you're looking for in your use-case is not clustering - it's a distance metric. When you get a new data point, you want to find the 3-5 most similar data points; there's no need for clustering for it. Calculate the distance from the new data point to each of the 'old' data points, and select the top 3-5. Now, which distance metric ...


1

It comes down to experimentation and two things: The total observations on the dataset Do you know how many categories? Sample size [<20] "Brich" is good on a small scale. [20,10k) Minibatch k-means, just to have a second option Hierarchical clustering. [>10k] k-means or Spectral clustering. Number of categories If you don't know how many ...


0

You can try Dynamic mode decomposition. Dynamic Time Warping. Found a nice resource on Towards data science blog. These two have proven better approaches than PCA for time series clustering. Happy coding đź’»


0

As you have the partial ground-truth (assuming for ALL clusters) I would suggest following a creative idea derived from Region Growing in image segmentation. As your clusters are imbalanced in number of the points thus density, they are probably captured by localy using DBSCAN. Run DBSCAN with different parameters and evaluate on capturing your ground-truth ...


2

So the question asks for other clustering algorithms which can be used in customer segmentation. In a similar way to using Affinity propagation to identify the "optimal" number of customer segments, to gain more control over the number of customer segments, you can also use agglomerative clustering (https://www.datanovia.com/en/lessons/...


1

You can do that. I propose the simplest one conditioned on the fact that number of data is not very large. In case you need more ideas please drop a comment. In this case, you can use the idea of similarity encoding based on Fuzzy String Matching and get the spectral embedding. The amount of data is crucial here as you need to do order of $n^2$ comparisons ...


0

The natural approach is to use a labelled dataset and a supervised learning technique. You can start with something simple, like using tf-idf for feature generation and train a simple logistic regression model. I think this is the first thing you should try, I see it more likely to succeed than the unsupervised techniques, and it is simple enough.


0

With purely one-hot encoded data this isn’t a problem. For example, the distance between a red square and a blue square in your second example (assuming you’re using Euclidean distance) is 1 in the red dimension and 1 in the blue dimension, so sqrt(1+1) overall (by pythagoras). Similarly, the distance between a red square and a red circle is 1 in the circle ...


0

I've not tried it, but it sounds like sklearn's Isomap might do the job, with metric='precomputed' and passing X as the distance matrix computed with Levenshtein/Jaccard/whatever. Have a look through the User Guide for the other manifold learning approaches, but Isomap stands out as applicable to me. See also https://stats.stackexchange.com/q/353298/232706


0

What @shepan6 said... but one other thing. Since you're grouping customers, you'll want to aggregate your dataset so that each row is a customer (not just a transaction) Your new columns might look like this, prior to your clustering exercise: customerid days_since_prior_transaction num_transactions_ever num_transactions_last_180_days num_online_sales ...


0

So the question is about how to before customer segmentation on this data. When I do any customer segmentation, I firstly think to myself, do I know how many segments prior to the analysis or not. If I do, Then I would use a clustering method like K-means clustering (https://towardsdatascience.com/understanding-k-means-clustering-in-machine-learning-...


0

The answer depends on many things, here are some questions and suggestions. it is fine to have a sparse matrix for clustering ! look into documentation of matrix decomposition here where the input can be sparse. here is an example using W for users or H for items, you can perform clustering. how much is the sparsity ? e.g. why not turning your data to ...


0

DTW per-group is the obvious answer. I've found DTW to be extremely computationally expensive, however. My favorite technique in the world is lesser-known/used, and would handle your problem nicely, however... and have the added benefit of being scalable. The downside to this technique is that it would only cluster on the "shape" of the time series,...


0

Label encoding is not a good idea if the nature of categories are not ordinal (it is actually not my favorite anyways). Use one-hot encoding and see how it works. You may apply a feature extraction on top of it, e.g. PCA, to reduce the noise coming from sparsity. The other idea is to label categories by their fraction in the feature, for example: [a,b,b,c,a,...


0

The best way to encode the data will be through any encoding mechanism like label encoder etc. But before handling the categorical variable check the correlation of a categorical variable with the target variable using the feature selection methods like chi square test with selectKbest.


2

First of all, you use two terms Clustering and Classification interchangably and I would like to draw your attention to this. Your problem is purely Clustering. Secondly, you asked for testing accuracy. As your problem is pure Clustering, there is no evaluation for that. The last but not least is the problem of "Short Text Understanding". In short ...


0

You can use k-means. It belongs do the domain of unsupervised learning, which means you do not have labels to test classification of examples. You can evaluate what the clusters, defined by k-means have in common and label them accordingly. In the k-means clustering you can assign the number of clusters(k) arbitrarily or evaluate the different number of ...


1

First of all, if you know that certain attributes shouldn't after the clusters, you should remove them altogether. There is no point in hoping that K-Means will figure it out on its own if that can be fixed upstream. Second, obviously, not every attribute should affect the clusters equally. K-Means is based on the concept of distances between your points. ...


0

So, the problem is how to cluster texts. Firstly, an alternative approach to representing document meaning, you can use Doc2Vec and compare similarities between document embeddings (https://medium.com/wisio/a-gentle-introduction-to-doc2vec-db3e8c0cce5e). Secondly, if you are unsure about the ideal number of clusters, instead of using k-means, you can use ...


0

If you want to group your dataset into k different group based on some feature( here checkout history) you can use K Means Clustering algorithm to cluster them into different groups. You will find sklearn k means clustering module helpful. All you need to do is provide the data into it and choose appropriate hyperparameters.


0

It is fourier transformation problem. All you need to do is to do a fourier transformation of your discrete signal and it will tell all the different frequencies it consists of. Numpy.fft class will be helpful.


0

I can't see all your points on the plot. It's possible to have a cluster with most points and one cluster with very few points since it is possible that most points are close to each other and we have another small clusters with only a few points. However, you might like to explore the possibility of increasing the number of clusters.


0

The simplest way is to build a matrix representing how often two items are bough together: $M[i,j]$ is the count of how many times items $i$ and $j$ are bought together (note that you need to fill only a diagonal matrix, since it's symmetric). From there you can calculate whatever you need: most frequent $B$ item bought with a given item $A$, top most ...


0

Hi Try Gaussian Mixture Model ,it similar to kmeans but differs in few ways in nutshell think of kmeans as hard clustering model where 1 sample is assigned to only one cluster whereas GMM is soft clustering technique that tells the density(probability) of the each Gaussian mixtures(consider this as the cluster) to that data point,you can get both the labels ...


Top 50 recent answers are included