Questions tagged [unsupervised-learning]

Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.

130 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
2answers
126 views

Cluster evolution over time

I have a dataset of transactional data with customer ID and I want to segment the dataset into groups using cluster analysis. I'm interested in following the evolution of each cluster over time, but ...
4
votes
1answer
36 views

Is there a paper accomplishing finding physical law from observation without premade perception, using machine learning?

For example: Isaac Newton finds law of universal gravitation just by looking a falling apple, without any premade perception of that phenomenon. Is it possible to accomplish that kind of discovery ...
4
votes
1answer
161 views

Probability for label correctness in semi-supervised learning

I am aware of the existence of semi-supervised learning approaches, such as the Ladder Network, where only a subset of the data is labeled. Are there any methods or papers which consider correctness ...
4
votes
2answers
2k views

Constrained k-means algorithms in R (must-link constraints)

I currently face an unsupervised learning task that is to be approaches using clustering. More specifically, it is a segementation task and hence there is some prior knowledge about a) the number of ...
3
votes
1answer
70 views

Sentiment analysis of tweets (Train model on a labelled dataset and use on some other unlabelled data)

I have a huge amount of tweets on a particular topic say 'ABC' and the data is not labelled. I want to perform multi-class sentiment analysis of these tweets. I tried many unsupervised clustering ...
3
votes
0answers
72 views

What are the data preprocessing steps required before running K-Modes?

I have a clustering task at hand. The data that I have contains only categorical variables. So, k-modes seemed like the best option. But I am not sure what are the data pre processing steps required ...
3
votes
1answer
168 views

What value can I gain by doing exploratory data analysis on features (and thus data) before doing clustering?

This might not be a very good question, but I would still ask if it's beneficial to do EDA before running a clustering algorithm? I understand that EDA helps us generate good and helpful insights ...
3
votes
0answers
49 views

How to remove noise using morphological filtering

I have two groups of dots that both contain noise between them: The line that separates the two groups in the picture is diagonal in shape. I tried to use morphological filtering on this image to ...
3
votes
0answers
3k views

Anomaly detection using k-means clustering in Python

I'm working on an anomaly detection task in Python. Datasets regard a collection of time series coming from a sensor, so data are timestamps and the relative values. In order to find anomalies, I'm ...
3
votes
2answers
182 views

Grouping already clustered data (with a pre-defined x and y)

I have an already clustered data set (I wanna keep my x and y), where there's clearly a small group of elements in the middle that don't follow the expected patterns. I can select them manually, but ...
3
votes
0answers
59 views

Determine the most important documents for supervised learning

I have somewhat of a general/high level question. Assume I'm doing supervised machine learning on some text data (tweets for example) and categorizing the documents to a certain taxonomy (multi-class ...
3
votes
0answers
377 views

Examples for predict.FAMD?

I am doing a study on unsupervised data with various categorical variables. So I have found the FactoMineR package to be really handy for this, particularly with the FAMD functions. I can get to a ...
3
votes
0answers
338 views

sLDA vs. LDA+Classifier

For simplicity, suppose we're looking at Yelp reviews of restaurants, and are trying to classify the restaurant by cuisine type (e.g. "Italian, Japanese," etc.). Lets also assume our data already a ...
3
votes
1answer
102 views

Finding dominating attributes with in the clusters generated

I am having a dataset of customers where each customer is represented as some feature vector and I am applying K-means algorithm to this dataset. On the basis of those features, I can abstract and ...
2
votes
1answer
43 views

Dendrogram: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

I am trying to plot a Dendrogram to cluster data but this error is stopping me. My datea is here. I first chose columns to work with: ...
2
votes
0answers
39 views

Intuition behind One Class SVM (Scholkopf)

I am trying to understand the intuition behind the idea of finding a hyperplane that separates the training data from the origin in the feature space. Why separation from origin with a hyperplane ...
2
votes
0answers
410 views

Dealing with categorical variables in Isolation Forest

Isolation Forest is widely used when dealing with outlier/anomaly detection when we have no labels. The theory behind is that making random split at random points and counting how many splits you do ...
2
votes
0answers
153 views

K-Means Clustering Profile Plot & Data Normalization

I am new to k-means clustering and I am working on a project on cryptoanalysis. I have a few questions and I hope to get some help here. I have four variables and my variables data values can range ...
2
votes
0answers
60 views

From unsupervised to supervised in fraud detection

I have a question. I am working on the fraud detection domain. And I have data from imports to the country. As you can get from the title, I have unsupervised data. I do not know that the record is ...
2
votes
1answer
30 views

Clustering Weekday Weekend Data and Multicollinearity

Hi I have data of weekday and weekend step counts in which I extracted metrics from them such as the wd steps, we steps, standard deviation of wd steps, standard deviation of we steps and so on... <...
2
votes
1answer
35 views

What's the good index to choose number of clusters so that obtained clusters are homogeneous?

I perform a clustering on one-dimensional dataset and I need a way to automatically decide what's the optimal number of clusters from $k \in \{2, 3, 4, 5, 6\}$. The number of observations to cluster ...
2
votes
1answer
157 views

Anomaly detection - relation between thresholds and anomalies

I'm developing an anomaly detection program in Python. Main idea is to create a new LSTM model every day, training it with the previous 7 days and predict the next day. Then, using thresholds, find ...
2
votes
0answers
17 views

Temporal outlier Analysis on sensor data

I am working to find anomaly/outliers in sensor data using unsupervised machine learning (without training dataset). I have around 20000 samples taken per minute of various sensors. I just need to ...
2
votes
0answers
353 views

Training detector without bounding box data

From what I can see most object detection NNs (Fast(er) R-CNN, YOLO etc) are trained on data including bounding boxes indicating where in the picture the objects are localized. Is there any model ...
2
votes
0answers
173 views

Exploratory analysis and feature engineering for time till failure prediction using sensor data of engines

I am trying to do some data exploration and analysis on a dataset of engine sensor readings. I would like to determine if the data I have is good enough to predict a time till failure and possibly ...
2
votes
1answer
674 views

Is it possible to make a label automatically in supervised learning(Machine Learning)?

My background knowledge: Basically, supervised learning is based on labeled data. Using the labeled data, the machine can study and determine results for unlabeled data. To do that, for example, if we ...
2
votes
0answers
3k views

Grid Search on Unsupervised Sklearn Clustering?

I am trying to use clustering algorithms in sklearn and am using Silhouette score with cosine similarity as a metric to compare different algorithms. My question is due to the varying hyperparameters ...
2
votes
0answers
119 views

Using classification to find the best support and confidence measure in associative rule mining

I have been trying the find the best support and confidence values for associative rules mining. I came across the following approach from an answer on Quora - Picking the "appropriate" values for ...
1
vote
0answers
30 views

Help why to apply PCA here

Lets say we have a dataset of 9 dimensional points and I want to apply k-means algorithm. I was studying an example where they apply PCA before fitting the data into the clustering algorithm. The ...
1
vote
0answers
31 views

How can realize the evaluation/validation of unsupervised models through unlabeled data?

I'm researching anomaly detection, which is nothing else than outliers detection on a set of time-series web servers access log data or network traffic. Recently I re-faced to following fundamental ...
1
vote
0answers
25 views

Difference between Q-learning and G-learning in Reinforcement Learning?

What is the difference between Q-learning and G-learning in Reinforcement Learning? Please explain with formulas. An example source: Instead of relying on a utility of consumption, we present G-...
1
vote
1answer
38 views

Hierarchical dirichlet process results

I am thinking about using hierarchical dirichlet process to model a patent dataset. I've seen that HDP uses a base distribution and assumes that every topic comes from that base distribution. The ...
1
vote
0answers
32 views

K-Medoid Clustering with Point Weights

I asked the same question at Cross Validated, here I implemented a K-Medoid clustering algorithm recently; I have a number of points $x_1, ..., x_n$ which have various properties and a distance ...
1
vote
0answers
18 views

Explanation of Excess Mass(EM)

I was researching on evaluation metrics to understand the performance of unsupervised anomaly detection algorithms and I came across this paper The author suggests that EM and MV based numerical ...
1
vote
1answer
29 views

Is it better to have one model with more categories or less with two for multi-label classification?

For classifying text into three classes question, complain and complements where each sample can have multi-labels (question and complains, question and complements): is it better to have one model ...
1
vote
1answer
32 views

How to properly train your Self-Organized Map?

I stumbled recently upon the Self-Organized Map, an ANN architecture used to cluster high dimensional data, while simultaneously imposing a neighbourhood structure on it. It's trained through a ...
1
vote
2answers
36 views

How do I select the “best” unsupervised machine learning algorithm to cluster my specific dataset?

I want to cluster a dataset without prior knowledge on the correct amount of clusters. For different algorithms (i.e. k-means, gmm...) I can iterate through different values and try to find the best ...
1
vote
1answer
62 views

How to get the probability/closeness of a sample belonging to a specific cluster?

I'm new to this so please let me know if my logic of comparing cosine similarity and k-means is incorrect I got a set of ...
1
vote
0answers
27 views

How do I approach this problem?

Let's say I have a dataset with multiple types of multiple ingredients ($salt_1$,$ salt_2$, etc). Each $n\text{-th}$ variation of each ingredient vs flavor may be represented by an $n \times k$ matrix ...
1
vote
2answers
34 views

What is the most effective unsupervised ML algorithm to use when outliers are present in data set?

I am analyzing a portfolio of about 225 stocks and have gotten data for each of them based on their "Price/Earnings ratio", "Return on Assets", and "Earnings per share growth". I would like to cluster ...
1
vote
0answers
24 views

Benchmarking unsupervised learning by second stage classifiers?

Suppose we have a dataset $\{x^{(i)}, y^{(i)}\}_{i = 1}^N$ where $x^{(i)} \in \mathbb{R}^n$ and $y^{(i)} \in \{0, 1\}$ for simplicity. Our main goal is to apply some unsupervised learning algorithm ...
1
vote
1answer
57 views

Clustering sequences of sentence embeddings

I have a sequence of events, right now I am not worried about their actual times, just the order. This is a sequence of web page views. I have modelled my data as the following, where each element ...
1
vote
0answers
17 views

Clustering and producing final results to find next best customer to target(Ranked)

I have a problem where I need to cluster customer data that has all possible attributes to identify the next potential customer who can succeed the last customer in terms of buying a certain product. ...
1
vote
0answers
27 views

How we compare two paragraphs of unlabelled dataset semantically (STS)?

Column representation: Unique_id | Text1 | Text2 Unique_id 0 Text1 public show for reynolds suspension of his coaching licence. portrait sir joshua reynolds portrait of omai will get a public airing ...
1
vote
2answers
168 views

How can we perform STS(Semantic Textual Similarity) on UnSupervised dataset using Deep Learning?

How to implement STS(Semantic Textual Similarity) on unlabelled dataset. Dataset column contains Unique_id, text1(contains paragraph), text2(contains paragraph). Ex: Column representation: Unique_id ...
1
vote
1answer
47 views

Univariate Outlier Detection

Let's say I have a dataset with the following format: customerid product orders_in_last7days orders_in_last6days orders_in_last5days orders_in_last4days orders_in_last3days orders_in_last2days ...
1
vote
0answers
29 views

Predict with some probability the day of the month paycheck received through daily transactions

I am using R to do machine learning. I have daily shopping expenditure data of individuals over a couple of months and my goal is to be able to identify through their shopping pattern the day of the ...
1
vote
0answers
39 views

Unsupervised Function Optimization using Input and Output for Loss Function?

I have some vectors {$\mathbf{X_1 ... X_n}$} and they are all of dimension 1 x N. Vectors {$\mathbf{X_1' ... X_n'}$} are also 1 x N and are related to {$\mathbf{X_1 ... X_n}$}, but the relation cannot ...
1
vote
0answers
18 views

Choosing a distance metric and a clustering algorithm for time series

For every entity I have a corresponding time series which is built by a sliding window (win_size=7d, win_shift=3d, so we have overlapped windows) With every win-shift, we count how many users are ...
1
vote
0answers
105 views

How to tune / choose the preference parameter of AffinityPropagation?

I have large dictionary of "pairwise similarity matrixes" that would look like the following: similarity['group1']: ...