Skip to main content

Questions tagged [unsupervised-learning]

Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.

157 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5 votes
0 answers
2k views

Dealing with categorical variables in Isolation Forest

Isolation Forest is widely used when dealing with outlier/anomaly detection when we have no labels. The theory behind is that making random split at random points and counting how many splits you do ...
Carlos Mougan's user avatar
3 votes
1 answer
32 views

Trim left tail of music in audio file

I have audio files, most of them start with the same music, and then a conversation begins. I want to trim the part of the music (which can be varied in length). I have no labels, I can transcribe the ...
David Harar's user avatar
3 votes
2 answers
629 views

What value can I gain by doing exploratory data analysis on features (and thus data) before doing clustering?

This might not be a very good question, but I would still ask if it's beneficial to do EDA before running a clustering algorithm? I understand that EDA helps us generate good and helpful insights ...
Akash Dubey's user avatar
3 votes
0 answers
195 views

How to remove noise using morphological filtering

I have two groups of dots that both contain noise between them: The line that separates the two groups in the picture is diagonal in shape. I tried to use morphological filtering on this image to ...
tamarlev's user avatar
3 votes
1 answer
105 views

Determine the most important documents for supervised learning

I have somewhat of a general/high level question. Assume I'm doing supervised machine learning on some text data (tweets for example) and categorizing the documents to a certain taxonomy (multi-class ...
John's user avatar
  • 31
3 votes
0 answers
593 views

Examples for predict.FAMD?

I am doing a study on unsupervised data with various categorical variables. So I have found the FactoMineR package to be really handy for this, particularly with the FAMD functions. I can get to a ...
I_Play_With_Data's user avatar
3 votes
0 answers
449 views

sLDA vs. LDA+Classifier

For simplicity, suppose we're looking at Yelp reviews of restaurants, and are trying to classify the restaurant by cuisine type (e.g. "Italian, Japanese," etc.). Lets also assume our data already a ...
Alex R.'s user avatar
  • 261
3 votes
1 answer
332 views

Finding dominating attributes with in the clusters generated

I am having a dataset of customers where each customer is represented as some feature vector and I am applying K-means algorithm to this dataset. On the basis of those features, I can abstract and ...
jaig's user avatar
  • 153
2 votes
1 answer
597 views

In Orange Data Mining, how do I use results from clustering a training-set to test and score a test-set?

I am performing analysis on the well-known 'Adult' data-set, available on UCI using Orange Data Mining. In a PhD thesis, Pelleg (2004; pg 79) uses unsupervised clustering of the prescribed training ...
Paul Higgins's user avatar
2 votes
0 answers
74 views

Multi-Class Document Classification with both known and un-known classes

Currently, I am building a multi-class document classifier which has to classify either 3 known classes, namely "Financial Report", "Insurance_Sheet", "Endorsement", and ...
Quan Nguyen Ha's user avatar
2 votes
1 answer
63 views

Tiering after clustering with Kmeans

I would like to have some suggestions on possible avenues that would make sense in the following context. 3 Optimal clusters have been identified in a 5000 list of customers using Kmeans Data model ...
Roger Steinberg's user avatar
2 votes
0 answers
23 views

What technique's can be used to identify and count individual animals in a dataset?

Problem: I have an image dataset that contains a lot of different chitals (a species of deer). The images are taken by cameratraps in a National Park. I would like to count the individual animals. For ...
hyilmaz's user avatar
  • 21
2 votes
0 answers
430 views

Unsupervised document similarity state of the art

I have a set of N documents with lengths ranging from 0 to more than 20000 characters. I want to calculate a similarity score between 0 and 1 between all pairs of documents where a higher number ...
user7017793's user avatar
2 votes
0 answers
60 views

unsupervised learning time series datasets

I experiment on building electricity power consumption datasets and try to see relationships of the power consumption with weather data and dummy variables that represent time-of-week. The only thing ...
bbartling's user avatar
  • 403
2 votes
1 answer
973 views

Dendrogram: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

I am trying to plot a Dendrogram to cluster data but this error is stopping me. My datea is here. I first chose columns to work with: ...
Sam.H's user avatar
  • 121
2 votes
0 answers
71 views

Intuition behind One Class SVM (Scholkopf)

I am trying to understand the intuition behind the idea of finding a hyperplane that separates the training data from the origin in the feature space. Why separation from origin with a hyperplane ...
batman's user avatar
  • 149
2 votes
0 answers
502 views

K-Means Clustering Profile Plot & Data Normalization

I am new to k-means clustering and I am working on a project on cryptoanalysis. I have a few questions and I hope to get some help here. I have four variables and my variables data values can range ...
Fabian Tan's user avatar
2 votes
2 answers
219 views

Semantic Search

There is a problem we are trying to solve where we want to do semantic search on our set of data, i.e we have a domain specific data (example: sentences talking about automobiles) Our data is just a ...
Farhaan Bukhsh's user avatar
2 votes
0 answers
101 views

From unsupervised to supervised in fraud detection

I have a question. I am working on the fraud detection domain. And I have data from imports to the country. As you can get from the title, I have unsupervised data. I do not know that the record is ...
gammazplaude's user avatar
2 votes
1 answer
58 views

What's the good index to choose number of clusters so that obtained clusters are homogeneous?

I perform a clustering on one-dimensional dataset and I need a way to automatically decide what's the optimal number of clusters from $k \in \{2, 3, 4, 5, 6\}$. The number of observations to cluster ...
jakes's user avatar
  • 95
2 votes
1 answer
512 views

Anomaly detection - relation between thresholds and anomalies

I'm developing an anomaly detection program in Python. Main idea is to create a new LSTM model every day, training it with the previous 7 days and predict the next day. Then, using thresholds, find ...
Giordano's user avatar
  • 345
2 votes
0 answers
25 views

Temporal outlier Analysis on sensor data

I am working to find anomaly/outliers in sensor data using unsupervised machine learning (without training dataset). I have around 20000 samples taken per minute of various sensors. I just need to ...
sdave1's user avatar
  • 21
2 votes
1 answer
669 views

How to tune / choose the preference parameter of AffinityPropagation?

I have large dictionary of "pairwise similarity matrixes" that would look like the following: similarity['group1']: ...
Mehdi's user avatar
  • 121
2 votes
0 answers
551 views

Training detector without bounding box data

From what I can see most object detection NNs (Fast(er) R-CNN, YOLO etc) are trained on data including bounding boxes indicating where in the picture the objects are localized. Is there any model ...
Rahul's user avatar
  • 21
2 votes
0 answers
191 views

Exploratory analysis and feature engineering for time till failure prediction using sensor data of engines

I am trying to do some data exploration and analysis on a dataset of engine sensor readings. I would like to determine if the data I have is good enough to predict a time till failure and possibly ...
broffesor_matt's user avatar
2 votes
0 answers
2k views

SOM initial values for learning rate and neighborhood sigma

I am using SOM (Self-Organizing Maps) of Kohonen, or more specifically, the MiniSom, found here to cluster and visualize my data. As you can see in the above site, the example given is: ...
passion's user avatar
  • 121
2 votes
0 answers
4k views

Grid Search on Unsupervised Sklearn Clustering?

I am trying to use clustering algorithms in sklearn and am using Silhouette score with cosine similarity as a metric to compare different algorithms. My question is due to the varying hyperparameters ...
GNMO11's user avatar
  • 121
2 votes
0 answers
36 views

With EM algorithm, can you infer the location and variance of each "peak" in a pdf? Gaussian Mixture Models?

When I plot my data into bins, there is a frequency of data points per bin, which I can plot with a histogram. Based on this probability density function, I would like to find the maximum likelihood ...
ShanZhengYang's user avatar
2 votes
0 answers
133 views

Using classification to find the best support and confidence measure in associative rule mining

I have been trying the find the best support and confidence values for associative rules mining. I came across the following approach from an answer on Quora - Picking the "appropriate" values for ...
kusur's user avatar
  • 129
2 votes
1 answer
97 views

Decision tree to get difference in rates in two groups?

I have two sample groups of customers, each customer has 100s of features. For a single sample, i would use Decision Trees to find sub-groups that have a high churn rate. Thats easy. However, my ...
Arslán's user avatar
  • 131
1 vote
0 answers
39 views

Can unsupervised pretraining (autoencoders) be used for u-nets?

TLDR: Will a u-net pretrained as an autoencoder be able to learn a latent representation of the data if the encoder weights are frozen (can't game the system and pass forward the unmodified image)? ...
user avatar
1 vote
0 answers
26 views

What algorithm should I use when trying to find closest record matches when records contain both categorical and discrete attributes?

I have 100000 records that have discrete features like topic (analytics etc) and categorical features like ticket details (eg: I need help with analytics for my business). When creating a new record, ...
Jack Smith's user avatar
1 vote
1 answer
1k views

Clustering for Sparse Data Matrix of high dimension

I currently have a dataset of 1000 entries with 512 features that are sparse. I want to cluster them. I have attempted using kmeans, but found that the clustering wasn't very good, and have been ...
Is land's user avatar
  • 11
1 vote
1 answer
21 views

Best Clustering Method for Dataset with Few Distinct Values

I have a dataset that has the opinions of 30 different TV shows for 2000 high school students. A student could have said they liked the show, did not have an opinion, or disliked the show. These ...
user138002's user avatar
1 vote
0 answers
51 views

Which clustering/partitioning algorithms can operate on arbitrary pairwise similarity or distance matrices?

I'm relatively new to cluster analysis, and I'm exploring options for general-purpose, non-hierarchical, strict partitioning of data based on a pre-computed $N\times N$ pairwise similarity matrix. ...
Grant Petty's user avatar
1 vote
0 answers
38 views

massively imbalanced data

I am dealing with time series data with +200K (every minute for 6 months)record of gas turbine I am trying to early detect the fault (0 or 1-fault). The issues with the data are: 1.the fault occurred ...
ahman's user avatar
  • 11
1 vote
1 answer
36 views

Can clustering results based on probability be used for supervised learning?

I'm a beginner and I have a question. Can clustering results based on probability be used for supervised learning? Manufacturing data with 80000 rows. It is not labeled, but there is information that ...
hahaha's user avatar
  • 11
1 vote
0 answers
24 views

How do I evaluate if my data represent the target variable before training a machine learning algorithm?

I have a dataset of points cloud where each point in the point cloud has a variable. I am trying to relate the local geometry features to that point variable by using FPFH, This means I am generating ...
Hamzalihi's user avatar
1 vote
1 answer
28 views

Partially labelled open-class classification problem with heavy overlap

Let's say we have a corpus of text, including discussions about movies and about sports. Unsupervised clustering would typically cluster into the two topics of discussion. However, we are interested ...
basiliskcompliantentity's user avatar
1 vote
0 answers
196 views

Unsupervized latent truth discovery on text data

I have text infomration from several different sources. I need to identify the most reliable source in an unsupervized manner (no labels about true/false or ground truth to train on available). I ...
Peter's user avatar
  • 7,526
1 vote
0 answers
21 views

Two sets of topics/words in Topic Modeling

In short, the question is: I have two sets of words per document. I would like to extract two sets of topics per document corresponding to sets of words. To be more precise: Document(d) can be ...
aloskam's user avatar
  • 11
1 vote
0 answers
37 views

Model doesn't know German well enough

I have a model that generates questions and answers based on input text. The texts are in German and based on observations it seems like the model doesn't know German well enough. I need to pretrain ...
futuredataengineer's user avatar
1 vote
1 answer
61 views

Clustering with hierarchical data dependencies

I am currently looking into how to cluster data with hierarchical dependencies. An example of a problem that I want to cluster: we would like to cluster cities to identify similar characteristics with ...
ml_ds_lm's user avatar
1 vote
0 answers
17 views

Clustering dataset with and without estimating means (no EM algorithm)

Given a dataset $D$ of the form $$ D = \{ (x_0,y_0), (x_1,y_1),\ldots,(x_{n},y_n) $$ sampled from a Gaussian mixture model with identity covariance matrices, I want to understand what are my options ...
Marion's user avatar
  • 133
1 vote
0 answers
363 views

Using STL(Seasonal-Trend decomposition using LOESS) for Anomaly detection

I am using STL to decompose my time series data in Season, trend and residual and then by applying this(see below) on residual. I am detecting the anomaly ...
Gaurav Saini's user avatar
1 vote
3 answers
162 views

Real-Time Outlier/Anomaly Detection?

My data is the usage/playing statistics for players of a specific game. One data point for a user is aggregated statistics for one week. The goal is: to detect when the player's account was stolen/...
newbieAtLife5741's user avatar
1 vote
0 answers
57 views

Unsupervised Learning with audio recordings

I had a (probably crazy) idea for a project and I was wondering if you all think it would be in any way possible. I'm interested in analyzing sounds made by different types of animals (for example ...
AffableAmbler's user avatar
1 vote
0 answers
20 views

Compute similiarty between labels

I have a labeled dataset and I created a duplicate of this dataset and removed the labels and applied K-means clustering with k= the number of labels in the original data set I want to compute ...
Mohamed Amine's user avatar
1 vote
0 answers
49 views

Help why to apply PCA here

Lets say we have a dataset of 9 dimensional points and I want to apply k-means algorithm. I was studying an example where they apply PCA before fitting the data into the clustering algorithm. The ...
batman's user avatar
  • 149
1 vote
1 answer
666 views

Difference between Q-learning and G-learning in Reinforcement Learning?

What is the difference between Q-learning and G-learning in Reinforcement Learning? Please explain with formulas. An example source: Instead of relying on a utility of consumption, we present G-...
develarist's user avatar