New answers tagged

0

Within each currency (say SGD or INR), you can divide the salary by quartiles, for e.g the top 25 percentiles of salaries within each currency can be assigned a value of 4, the next 25th percentile a 3 and so and so forth. This way you are able to normalize salaries across currencies.


0

The usual way is to present the top N (e.g. top 10) words for the cluster: With distance-based clustering like K-means, the top words can be picked as the closest ones to the centroid. With probabilistic methods such as LDA, the top words are the ones with the highest probability for the topic.


0

Some common techniques to reduce number of features: Missing Values Ratio. Data columns with too many missing values are unlikely to carry much useful information. Thus data columns with number of missing values greater than a given threshold can be removed. The higher the threshold, the more aggressive the reduction. Low Variance Filter. Similarly to the ...


0

Yes, you could start with k-means on the 100-dim data. Make sure you have normalized your data to perhaps zero mean and unit variance before running k-means. Use the elbow method (google it) to determine the optimal number of clusters for your data. By the way you also want to check if the two dimensions are correlated in any way. If it is then you can use ...


1

One option would be look at conditional probability based on percentiles. First, find percentiles based on all the data. For example, 99th percentile is 1432 milliseconds. Then, find percent of a specific user request above that threshold. For example U1 has 50% of requests above 99th percentile. This could be made into a cross-tabs table for easier ...


2

DBSCAN is the algorithm of choice for this task. This a density based algorithm which will look for clusters according to two main parameters, epsilon and min samples. It will also identify those samples that do not form any cluster according to prior parameters as outliers. From Scikit-learn documentation: The DBSCAN algorithm views clusters as areas of ...


0

It depends how you want to cluster the data, but here are some options.... FEATURE ENGINEERING You could, for example, completely ignore the timestamps and just seek to cluster the different modes of operation are based on the magnitude of the feature alone. Here, simply extract the values into a new feature list. Otherwise, you have to consider what is ...


0

You first create inverted indexes or postings list. Then, using the term frequency and document frequency you calculate tf idf with the formula $tf* log({N \over df})$. For more details check this blog.


0

When you change the scale or range of your features the euclidean distance between pairs of observations in your dataset changes significantly and one feature with the larger range dominates over all the others. So, in your case, k-means was tricked into believing that the depth in metres is more important than the other feature with the lower scale. You can ...


1

The intuition built by the top response is spot-on for tf-idf vectors, and carries over to any vector that naturally wants to be normalized. However, in such circumstances, cosine similarity is bijective with Euclidean distance, so there's no real advantage to one over the other theoretically; in practice, cosine similarity is faster then. The second ...


1

Good points by shepan6. It is also worth to mention about feature extraction techniques to lower the number of features. You can use common feature extraction techniques such as PCA or eigen projections to transform the data into a lower dimensions. After lowering the number of features like this you can appply techniques such as clustering


0

This is a good question! The first thing to ask yourself is what is the purpose of carrying out clustering over this dataset? (e.g. to identify certain customer groups, by clustering them into clusters which represent how much a customer group spends on average) This will help with selecting appropriate features. In the case of identify clusters with similar ...


1

This happens if: There is no clear way to separate the clusters, i.e. there is only one large group of instances and the instances which are far from this group are too far from each other to form their own cluster. The linkage criterion plays a big role: this is more likely to happen with single linkage clustering , because as the main cluster grows it ...


0

Clustering doesn't work like this: for example k-means assigns an instance to the closest centroid, and since there is always a closest centroid there is a always a cluster that an instance "belongs to". So you need a different approach if you plan to have the possibility of in instance "not in any group": redo the clustering on the full ...


0

One approach is to do dimensionality sampling, that is drop some features and see the resulting dataset that arises. If there is some objective importance metric (eg Correlation, PCA) that quantifies the 2 features as more important, you can try that directly. Else one can try iteratively to drop features and test the resulting dataset. This approach does ...


0

If education level and income are more important than other features, you can multiply those features by a factor (greater than 1). That will allow the clustering algo to focus more on those features than the rest. For larger datasets, with features where the differences are not so obvious, you will need to rely on your judgement based on the end objective ...


1

I was wondering the same thing, then I tried something very logical. I experimented with both the cases. Here's a nice clustering plot, with round clusters, with scaling: Here's the clearly skewed clustering plot, one without scaling! In the second plot, we can see 4 vertical planar clusters. Clustering algorithm k-means is completely dominated by the ...


0

There are different approaches from easy to complicated Similarity-Based Distance or similarity matrix is calculated for different topics (forums) and if two topics are more similar than a threshold, you suggest them to the users of the other one. For this, you need to gather all texts of different forums and vectorise them (from TF-IDF to Neural Embedding)....


0

I am not sure that the positions of the force-directed graph perform better than direct clustering on the original data. A typical clustering approach when you have a distance matrix is to apply hierarchical clustering. With scikit-learn, you can use a type of hierarchical clustering called agglomerative clustering, e.g.: from sklearn.cluster import ...


0

I am not sure I fully understand your question, but generally the metric you use to build a DAG needs to be understood in terms of how you interpret relevant results. That said, a cluster map sounds like a good match for your use case. That is, a correlation matrix with sorted values according to linkage clustering on your datapoints. See below an example: ...


1

You can get a dendrogram from any hierarchical clustering method. The tricky thing here is how to compute the distances between the words. If efficiency is your main concern, I would consider using HDBSCAN clustering. The Jaro-Winkler distance was originally designed for such tasks. There is an efficient implementation in the python Levenshtein package, but ...


Top 50 recent answers are included