Questions tagged [unsupervised-learning]
Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.
164
questions with no upvoted or accepted answers
5
votes
0
answers
2k
views
Dealing with categorical variables in Isolation Forest
Isolation Forest is widely used when dealing with outlier/anomaly detection when we have no labels. The theory behind is that making random split at random points and counting how many splits you do ...
- 6,052
3
votes
1
answer
29
views
Trim left tail of music in audio file
I have audio files, most of them start with the same music, and then a conversation begins. I want to trim the part of the music (which can be varied in length). I have no labels, I can transcribe the ...
- 151
3
votes
1
answer
574
views
What value can I gain by doing exploratory data analysis on features (and thus data) before doing clustering?
This might not be a very good question, but I would still ask if it's beneficial to do EDA before running a clustering algorithm?
I understand that EDA helps us generate good and helpful insights ...
- 676
3
votes
0
answers
165
views
How to remove noise using morphological filtering
I have two groups of dots that both contain noise between them:
The line that separates the two groups in the picture is diagonal in shape.
I tried to use morphological filtering on this image to ...
- 43
3
votes
1
answer
88
views
Determine the most important documents for supervised learning
I have somewhat of a general/high level question.
Assume I'm doing supervised machine learning on some text data (tweets for example) and categorizing the documents to a certain taxonomy (multi-class ...
- 31
3
votes
2
answers
748
views
Is it possible to make a label automatically in supervised learning(Machine Learning)?
My background knowledge: Basically, supervised learning is based on labeled data. Using the labeled data, the machine can study and determine results for unlabeled data. To do that, for example, if we ...
- 31
3
votes
0
answers
527
views
Examples for predict.FAMD?
I am doing a study on unsupervised data with various categorical variables. So I have found the FactoMineR package to be really handy for this, particularly with the FAMD functions. I can get to a ...
- 2,079
3
votes
0
answers
429
views
sLDA vs. LDA+Classifier
For simplicity, suppose we're looking at Yelp reviews of restaurants, and are trying to classify the restaurant by cuisine type (e.g. "Italian, Japanese," etc.). Lets also assume our data already a ...
- 261
3
votes
1
answer
275
views
Finding dominating attributes with in the clusters generated
I am having a dataset of customers where each customer is represented as some feature vector and I am applying K-means algorithm to this dataset. On the basis of those features, I can abstract and ...
- 153
2
votes
1
answer
324
views
In Orange Data Mining, how do I use results from clustering a training-set to test and score a test-set?
I am performing analysis on the well-known 'Adult' data-set, available on UCI using Orange Data Mining. In a PhD thesis, Pelleg (2004; pg 79) uses unsupervised clustering of the prescribed training ...
- 121
2
votes
0
answers
51
views
Multi-Class Document Classification with both known and un-known classes
Currently, I am building a multi-class document classifier which has to classify either 3 known classes, namely "Financial Report", "Insurance_Sheet", "Endorsement", and ...
2
votes
1
answer
56
views
Tiering after clustering with Kmeans
I would like to have some suggestions on possible avenues that would make sense in the following context.
3 Optimal clusters have been identified in a 5000 list of customers
using Kmeans
Data model ...
- 111
2
votes
0
answers
21
views
What technique's can be used to identify and count individual animals in a dataset?
Problem:
I have an image dataset that contains a lot of different chitals (a species of deer). The images are taken by cameratraps in a National Park. I would like to count the individual animals.
For ...
- 21
2
votes
0
answers
420
views
Unsupervised document similarity state of the art
I have a set of N documents with lengths ranging from 0 to more than 20000 characters. I want to calculate a similarity score between 0 and 1 between all pairs of documents where a higher number ...
- 21
2
votes
0
answers
54
views
unsupervised learning time series datasets
I experiment on building electricity power consumption datasets and try to see relationships of the power consumption with weather data and dummy variables that represent time-of-week.
The only thing ...
- 383
2
votes
1
answer
959
views
Dendrogram: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()
I am trying to plot a Dendrogram to cluster data but this error is stopping me. My datea is here.
I first chose columns to work with:
...
- 21
2
votes
0
answers
66
views
Intuition behind One Class SVM (Scholkopf)
I am trying to understand the intuition behind the idea of finding a hyperplane that separates the training data from the origin in the feature space.
Why separation from origin with a hyperplane ...
- 139
2
votes
0
answers
463
views
K-Means Clustering Profile Plot & Data Normalization
I am new to k-means clustering and I am working on a project on cryptoanalysis. I have a few questions and I hope to get some help here.
I have four variables and my variables data values can range ...
- 21
2
votes
2
answers
184
views
Semantic Search
There is a problem we are trying to solve where we want to do semantic search on our set of data,
i.e we have a domain specific data (example: sentences talking about automobiles)
Our data is just a ...
2
votes
0
answers
87
views
From unsupervised to supervised in fraud detection
I have a question. I am working on the fraud detection domain. And I have data from imports to the country. As you can get from the title, I have unsupervised data. I do not know that the record is ...
- 21
2
votes
1
answer
55
views
What's the good index to choose number of clusters so that obtained clusters are homogeneous?
I perform a clustering on one-dimensional dataset and I need a way to automatically decide what's the optimal number of clusters from $k \in \{2, 3, 4, 5, 6\}$. The number of observations to cluster ...
- 95
2
votes
1
answer
447
views
Anomaly detection - relation between thresholds and anomalies
I'm developing an anomaly detection program in Python.
Main idea is to create a new LSTM model every day, training it with the previous 7 days and predict the next day.
Then, using thresholds, find ...
- 335
2
votes
0
answers
21
views
Temporal outlier Analysis on sensor data
I am working to find anomaly/outliers in sensor data using unsupervised machine learning (without training dataset). I have around 20000 samples taken per minute of various sensors. I just need to ...
- 21
2
votes
1
answer
527
views
How to tune / choose the preference parameter of AffinityPropagation?
I have large dictionary of "pairwise similarity matrixes" that would look like the following:
similarity['group1']:
...
- 121
2
votes
0
answers
526
views
Training detector without bounding box data
From what I can see most object detection NNs (Fast(er) R-CNN, YOLO etc) are trained on data including bounding boxes indicating where in the picture the objects are localized.
Is there any model ...
- 21
2
votes
0
answers
190
views
Exploratory analysis and feature engineering for time till failure prediction using sensor data of engines
I am trying to do some data exploration and analysis on a dataset of engine sensor readings. I would like to determine if the data I have is good enough to predict a time till failure and possibly ...
2
votes
0
answers
2k
views
SOM initial values for learning rate and neighborhood sigma
I am using SOM (Self-Organizing Maps) of Kohonen, or more specifically, the MiniSom, found here to cluster and visualize my data.
As you can see in the above site, the example given is:
...
- 121
2
votes
0
answers
4k
views
Grid Search on Unsupervised Sklearn Clustering?
I am trying to use clustering algorithms in sklearn and am using Silhouette score with cosine similarity as a metric to compare different algorithms. My question is due to the varying hyperparameters ...
- 121
2
votes
0
answers
130
views
Using classification to find the best support and confidence measure in associative rule mining
I have been trying the find the best support and confidence values for associative rules mining. I came across the following approach from an answer on Quora -
Picking the "appropriate" values for ...
- 129
2
votes
1
answer
87
views
Decision tree to get difference in rates in two groups?
I have two sample groups of customers, each customer has 100s of features. For a single sample, i would use Decision Trees to find sub-groups that have a high churn rate. Thats easy.
However, my ...
- 131
1
vote
0
answers
15
views
What algorithm should I use when trying to find closest record matches when records contain both categorical and discrete attributes?
I have 100000 records that have discrete features like topic (analytics etc) and categorical features like ticket details (eg: I need help with analytics for my business). When creating a new record, ...
- 11
1
vote
1
answer
127
views
Encode categorical data for unsupervised learning
What is the best encoder for categorical data in unsupervised learning?
I am using unsupervised learning on mixed data (such as K-means).
Before running my unsupervised algorithm, I am using dimension ...
- 11
1
vote
1
answer
830
views
Clustering for Sparse Data Matrix of high dimension
I currently have a dataset of 1000 entries with 512 features that are sparse. I want to cluster them. I have attempted using kmeans, but found that the clustering wasn't very good, and have been ...
- 11
1
vote
1
answer
21
views
Best Clustering Method for Dataset with Few Distinct Values
I have a dataset that has the opinions of 30 different TV shows for 2000 high school students.
A student could have said they liked the show, did not have an opinion, or disliked the show. These ...
- 11
1
vote
0
answers
43
views
Which clustering/partitioning algorithms can operate on arbitrary pairwise similarity or distance matrices?
I'm relatively new to cluster analysis, and I'm exploring options for general-purpose, non-hierarchical, strict partitioning of data based on a pre-computed $N\times N$ pairwise similarity matrix. ...
- 111
1
vote
0
answers
33
views
massively imbalanced data
I am dealing with time series data with +200K (every minute for 6 months)record of gas turbine
I am trying to early detect the fault (0 or 1-fault). The issues with the data are:
1.the fault occurred ...
- 11
1
vote
1
answer
35
views
Can clustering results based on probability be used for supervised learning?
I'm a beginner and I have a question.
Can clustering results based on probability be used for supervised learning?
Manufacturing data with 80000 rows. It is not labeled, but there is information that ...
- 11
1
vote
0
answers
24
views
How do I evaluate if my data represent the target variable before training a machine learning algorithm?
I have a dataset of points cloud where each point in the point cloud has a variable. I am trying to relate the local geometry features to that point variable by using FPFH, This means I am generating ...
- 11
1
vote
1
answer
24
views
Partially labelled open-class classification problem with heavy overlap
Let's say we have a corpus of text, including discussions about movies and about sports. Unsupervised clustering would typically cluster into the two topics of discussion. However, we are interested ...
1
vote
0
answers
108
views
Unsupervized latent truth discovery on text data
I have text infomration from several different sources. I need to identify the most reliable source in an unsupervized manner (no labels about true/false or ground truth to train on available). I ...
- 7,297
1
vote
0
answers
19
views
Two sets of topics/words in Topic Modeling
In short, the question is: I have two sets of words per document. I would like to extract two sets of topics per document corresponding to sets of words.
To be more precise:
Document(d) can be ...
- 11
1
vote
0
answers
34
views
Model doesn't know German well enough
I have a model that generates questions and answers based on input text. The texts are in German and based on observations it seems like the model doesn't know German well enough.
I need to pretrain ...
1
vote
1
answer
56
views
Clustering with hierarchical data dependencies
I am currently looking into how to cluster data with hierarchical dependencies. An example of a problem that I want to cluster: we would like to cluster cities to identify similar characteristics with ...
- 11
1
vote
2
answers
102
views
unsupervised anomaly detection for univariate fast frequency time series data?
I have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer
This time series ...
- 1,794
1
vote
0
answers
17
views
Clustering dataset with and without estimating means (no EM algorithm)
Given a dataset $D$ of the form
$$
D = \{ (x_0,y_0), (x_1,y_1),\ldots,(x_{n},y_n)
$$
sampled from a Gaussian mixture model with identity covariance matrices, I want to understand what are my options ...
- 133
1
vote
0
answers
307
views
Using STL(Seasonal-Trend decomposition using LOESS) for Anomaly detection
I am using STL to decompose my time series data in Season, trend and residual and then by
applying this(see below) on residual. I am detecting the anomaly
...
- 11
1
vote
1
answer
1k
views
Clustering on binary data
I am working on clustering on binary data which has 25 features,
sample
Feature 1
Feature 2
Feature 3
......
Feature 25
1
1
0
0
011101
1
2
0
1
0
010011
0
3
1
0
1
101001
1
and I have used the ...
- 11
1
vote
3
answers
105
views
Real-Time Outlier/Anomaly Detection?
My data is the usage/playing statistics for players of a specific game. One data point for a user is aggregated statistics for one week. The goal is to be able to detect when the account of the player ...
1
vote
0
answers
44
views
Unsupervised Learning with audio recordings
I had a (probably crazy) idea for a project and I was wondering if you all think it would be in any way possible. I'm interested in analyzing sounds made by different types of animals (for example ...
- 363
1
vote
1
answer
89
views
An Unsupervised learning method suitable for large categorical data sets
I want to detect anomalies in the bank data set in an unsupervised learning method. However, in the bank data set, all columns except time and amount were categorical data, and about half of them had ...
- 11