6

Answering your question: yes, depending on the hyperparameters you choose, you could overfit the considered normal data, if you fit your separating hyperplane between normal and novel points being too much "shaped" on your input data. There are, for instance in case of one-class support vector machines, some important hyperparams like nu or gamma: ...


4

A detailed answer would require many pages of explanation, but I think a brief answer may point to the right direction for further research. First of all the choice of dimensionality reduction algorithm depends on the problem and data at hand. There is no golden standard. Your problem requirements dictate the best option(s) to try out. The main concept of ...


3

You can see this comparison table in sklearn, which gives some intuition about where and when each algorithm is successful: It might be a good idea to try both and evaluate their accuracy, with an unsupervised clustering metric, like the silhouette score, to get an objective measure of their performance on a specific dataset. Some other major differences ...


3

Unsupervised Learning is a collection of tools and you can use those tools for many purposes. Sometime, it's just for data visualisation / data exploration (see UMAP for example), to better understand the problem and get the idea how to tackle it. Sometimes it will lead to business decision like building processes for a specific cluster or building submodels ...


3

I'll suggest to test the sentence or the tweet for polarity. This can be done using the textblob library. It can be installed as pip install -U textblob. Once the text data polarity is found, it can be assigned as a separate column in the dataframe. Subsequently, the sentence polarity can then be used for further analysis. Polarity and Subjectivity are ...


2

It appears to me that what you're looking for in your use-case is not clustering - it's a distance metric. When you get a new data point, you want to find the 3-5 most similar data points; there's no need for clustering for it. Calculate the distance from the new data point to each of the 'old' data points, and select the top 3-5. Now, which distance metric ...


2

One reason why you aren't getting fitted values close to the true values could be the initial values of the parameters used. It's likely what you have found is a local maxima. You have to try a number of initial starts and then pick the one with that gives the highest likelihood.


2

I don't think so, not directly. SHAP is trying to explain each feature's effect on the prediction, but you have no label here. It might be better to ask therefore, what are you trying to explain? In the case of an isolation forest, you can find the short path through the trees to any anomaly. That path tells you why the trees separated it, based on what ...


2

You can do both, meaning a) look for Supervised Learning to empower my Data Insights b) make decision as a human/team/committee to do certain things based on those What does that mean? in the case of a) you can think of unsupervised learning as representation learning-meaning you learn the best quantitve represenation of your data, and than right at the end, ...


2

To add to WBM great citation, you should use K-means over Agglomerative when your final objetive is to use the trained algorithm to make inference over new unseen observations. I will try to illustrate this with an example: Imagine you have 2 models kmeans and aggcls both have been trained on data that correspond to information of customers on an specific ...


2

DBSCAN will always mark noisy points according to epsilon and min_samples parameters, so there is no way to avoid that unless you have very compact and "well defined" clusters, what seems unlikely.The short answers will be to use another clustering algorithm such as gaussian mixture, k-means, or Birch If your problem really needs you to use DBSCAN, ...


1

No need for machine learning here. After you've transposed the dataframe, just count the number of unique combinations in the new column, and then rank them by frequency. Set a suitable threshold of "rareness" (like freq=2 below) and you will have your list of strange combinations. There's a tool in Pandas for this called df.values_count() e.g. ...


1

IsolationForest doesn't work on Euclidean distance. Hence [0,0] is almost as good as [100,100] It builds random Trees on the dataset and expects that the Outlier will singled-out very early in the Tree while the Inliers will go deep. With that logic, it can figure out the Outlier. The IsolationForest ‘isolates’ observations by randomly selecting a feature ...


1

Clustering and recommendation are similar tasks, however in recommendation you usually want to recommend several items while clustering usually assigns each sample to only one cluster. Anyway for your problem a clustering or even a classifier might help. If labels are assigned on the basis of a similarity metric (and you have a good guess of what this metric ...


1

I might misunderstand something but it looks to me like you're trying to find a complex method for a simple problem: if there are many strings which occur multiple times in the list, you should deduplicate the list before comparing all the pairs. You could use a set, but since you will need to count how frequent each string is you should probably directly ...


1

GANs have many known problems. The main ones are: Lack of convergence. Vanishing gradients when discriminator is "too good", leading to stagnation of the generator. Mode collapse: the diversity of the generated samples tends to be very low, generating always the same values. GANs for image generation have been studied extensively. Other domains, ...


1

1- You better start with Isolation forest Isolation Forest This is a very simple algorithm where you can control the contamination rate of your data. 2- For visualization you can plot the anomalous points in red, and you can distinguish them using the Isolation Forest predict(X) function that returns -1 for outliers and 1 for the rest. 3- You can use LSTM by ...


1

This basically boils down to the Vanishing Gradient Problem. It is a well known issue that vanishing gradients limit the ability of deep neural nets to learn. And playing the min-max-game for GANs can lead to this issue for G as well: When you start to train a GAN G will not produce very good results, i.e. D can easily classify these as fake or real with a ...


1

You are absolutely right. This method is called customer segmentation. Here we cluster customers based on many features like their demographic and income. Suppose we get to know a particular set of people who belong to high income group/rich demographic are not spending much then we can do promotions to increase sales to get potential new customers hence it ...


1

If you want to identify the distance between MSAs. Then yes, I think it would be best to first aggregate your features such that each instance (row) represents an MSA. From there you will have an $n\times m$ matrix where $n$ is the number of MSA, and $m$ is the number of features you end up with. You can then apply your clustering algorithm, there are many ...


1

I'm not sure I understand your problem very well but let's see. First let me try to formalize the task as a ML problem: Identifying the SMS of interest is a binary classification task. Your "completeness" score seems to correspond to the standard recall measure. It is usually a good idea to also look at precision, i.e. out of the SMS identified as ...


1

This is basic architecture of spam filter : Statistically,spam bear lower entropy ( i.e., higher similarities) than legitimate emails. We could use bisect k-means clustering after doing topic modelling. In k-means we had to specify k which lead to drastic change in results and it also leads to empty clusters. I would recommend going through this paper as it ...


1

First of all, if you know that certain attributes shouldn't after the clusters, you should remove them altogether. There is no point in hoping that K-Means will figure it out on its own if that can be fixed upstream. Second, obviously, not every attribute should affect the clusters equally. K-Means is based on the concept of distances between your points. ...


1

I understand that you are trying to derive new informative feature from the available tweet texts. And you do it in two steps: first you calculate dummy binary features, next you want to aggregate all binary features into one numerical feature. Several aggregation rules come to mind: simply calculate the sum of all binary features (and multiply by -5 if you ...


1

Manually assigning a value to a feature level can be done. However, it is often better to allow the machine learning algorithm to learn the importance of different features during the training process. The general machine learning process starts with labeled data. If the labels are numeric, it is a regression problem. In the specific case of fake tweets, a ...


1

Try derivative after either a low-pass filter or smoothing (probably exponential smoothing) to cut down on the noise. Big changes result in a big derivative (up or down).


1

Is using unsupervised learning to setup supervised classification reasonable? Absolutely. This is a common strategy in ML. As you said yourself, using information coming from the data itself has the benefit of being less biased. Would it even also be possible to use unsupervised learning to create the labels? Technically yes. Though, some clustering ...


1

It is not the case that all data resembles some manifold, for most reasonable meanings of the phrase "resembles some manifold". Mathematically, zero dimensional manifolds are collections of points, and technically speaking all finite data sets can be thought of as zero dimensional manifolds. However, I'm quite sure that's not what you had in mind when you ...


1

Often people confuse between unsupervised feature selection (UFS) and dimensionality reduction (DR) algorithms as the same. For instance, a famous DR algorithm is Principal Component Analysis (PCA) which is often confused as a UFS method! Researchers have suggested that PCA is a feature extraction algorithm and not feature selection because it transforms the ...


1

You might be interested in this paper and python implementation of various other feature selection for clustering tools and papers: http://www.public.asu.edu/~huanliu/papers/pakdd00clu.pdf https://github.com/danilkolikov/fsfc An excerpt sumarizing the approach: We address the problem of selecting a subset of important features for clus tering for the whole ...


Only top voted, non community-wiki answers of a minimum length are eligible