Questions tagged [unsupervised-learning]
Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.
450
questions
0
votes
0
answers
30
views
Machine learning can be broken down into supervised, unsupervised, and reinforcement learning. Is there anything else?
Is it even logically possible to have a type of machine learning that doesn't fall into those three paradigms?
Supervised: a dataset of inputs and outputs are fed to an algorithm which learns a ...
0
votes
0
answers
12
views
Clustering of two datasets in different years
I want to analyze two datasets by running a clustering algorithm on both and comparing the results. The two datasets have the same variables. The only difference is that one dataset is from 2010 and ...
0
votes
0
answers
19
views
How to process dictionary of 2d-arrays with River?
I want to perform non-supervised anomaly detection of water quality spectra (2d-array: intensity vs wavelength) using incremental River python library.
As far as I have seen, the input data of an ...
- 1
0
votes
0
answers
14
views
Ranking risky routes
i'm looking to get the top 10 rank of the most dangerous routes. I have a routes table where each row is a route and it has features such as
avg daily traffic for past three years
No. of times where ...
- 1
1
vote
1
answer
31
views
Removing specific phrases from textual data (R)
I have the following reddit posts and I would like to clean the posts and remove from the data the specific phrase "Click to expand", while keeping all other words within a post the same.
<...
- 127
0
votes
0
answers
15
views
Unsupervised rule extraction of categorical data
I have a dataset of network traffic with three features that I would like to extract rules from in order to apply firewall/flow control rules i.e. the permitted flows.
I am able to classify a ...
0
votes
1
answer
39
views
9-dimensional data classification
I have a dataset with 9 columns and 20 million rows. I have now split the dataset into two parts using IQR.
One part is my inliers, the other part is my outliers.
I now want to build a categorizer ...
1
vote
1
answer
72
views
How to Justify Anomalies Detected by Unsupervised Anomaly Detection Models? [closed]
I'm working on an unsupervised anomaly detection project involving a large sensor dataset, where I aim to identify anomalies without the aid of labeled data. While I've implemented several ...
0
votes
1
answer
28
views
Overfitting in the trained model
For my project on the classification problem of predicting churn customers, I trained various base models using k-fold validation on the training dataset and out of which random forest gave the best ...
0
votes
0
answers
15
views
How to update the weights of a ranking algorithms through unsupervised learning?
I have, let's say, 100 samples with n features, and I want to rank them by learning the weights for each feature. Weights must be learned from implicit ratings from the user (eg. user click). However, ...
0
votes
1
answer
32
views
Different scaling methods of different features results in a faux dependency between them
My dataset contains the following two features: "movie duration" (minutes) and "tv shows duration" (seasons). If a certain sample is of type "movie", it's duration will ...
- 1
0
votes
1
answer
81
views
Clustering task: drop or not drop a categorical attribute/feature for which each row in the dataset contains a different value
I am dealing with a clustering task. In the dataset I am using there is a categorical feature and for each row in the dataset I have a different value for that feature (my dataset consists of 1000 ...
- 1
0
votes
0
answers
11
views
What kind of learning do I need ? (use-case specific)
Consider a scenario where I have a model trained on gesture videos (say a 3D ResNet). I am looking for a technique (or a combination) that allows me to further train the model every time I have a new ...
- 149
0
votes
0
answers
22
views
Using Autoencoder in Python for Data Selection and Risk Level Calculation
I have a dataset that includes 100 data points from each sensor, representing various measurements. These measurements can be used to calculate the level of risk associated with each sensor. However, ...
- 1
0
votes
0
answers
46
views
Equivalence between K-Means objective functions proof
I am working on a project involving the K-Means clustering algorithm, and I am trying to prove the equivalence between different formulations of the objective function. Specifically, I want to show ...
0
votes
0
answers
18
views
Unable to reproduce result for CIFAR10 with ResNet50 backbone and SIMCLR self-supervised algorithm
I am recently working on self-supervised learning, particularly simclr paradigm. I found it hard to reproduce results on cifar10 with linear evaluation. I now get round 60%, while the paper reports 80%...
- 1
2
votes
1
answer
74
views
How to improve the preservation of the global data structure in UMAP?
I have a dataset, where the features are comprised of points arranged in a regular grid on a simplex. Each of these points are defined as follows: A point $\mathbf{x}$ on the simplex can be ...
- 121
0
votes
0
answers
21
views
Can TSNE and other visualisation methods separate multivariate normal blobs?
Consider we have two classes of points. Both of them come from a multivariate normal distribution with an unrestricted covariance matrix.
Let's assume, that the densities of those distributions do not ...
0
votes
1
answer
351
views
Rigth way to find Lorentzian distance between 2 point
Following this paper and this paper, I'm trying to implement the formula for the Lorentzian distance between 2 points (aka the distance between 2 points in Lorentzian space). I'll use this a the ...
1
vote
0
answers
15
views
What algorithm should I use when trying to find closest record matches when records contain both categorical and discrete attributes?
I have 100000 records that have discrete features like topic (analytics etc) and categorical features like ticket details (eg: I need help with analytics for my business). When creating a new record, ...
- 11
0
votes
0
answers
64
views
How to use IsolationForest for anomaly detection of a 1D-array, given given another feature?
[Beginner at ML]
Hi,
I would like to use unsupervised anomaly detection of spectrograms.
Currently, I am trying IsolationForest on a bunch of 1D-arrays (light intensity vs wavelength) in order to ...
- 1
0
votes
0
answers
24
views
Varying feature vector lengths for learning
[I am a total beginner in machine learning algorithms]
I have 10 spectrograms (lines) for phytoplankton (each composed of 288 points).
Each spectrogram is associated with a phytoplankton dendity data ...
- 1
0
votes
0
answers
24
views
Can sigmoid function applied on distance calculation of Self Organizing Map improve the accuracy?
I just thought how sigmoid function helps logistic regression making more accurate classification and wonder if it can be applied on distance calculation of Self Organizing Map and making more better/...
- 1
0
votes
0
answers
8
views
How do you know which examples of data are the most accurate classified using Self Organizing Map?
Here is an example of data where I apply the SOM general rule:
If distance1 > distance2 than class C1 else class C2 ; So in this example I use a self organizing map which clusters only two classes ...
- 1
0
votes
2
answers
1k
views
Why is UMAP used in combination with other Clustering Algorithm?
I've noticed that UMAP is often used in combination with other clustering algorithms, such as K-means, DBSCAN, HDBSCAN. However, from what I've understood, UMAP can be used for clustering tasks. So ...
- 103
0
votes
1
answer
15
views
Unsupervised ranking of samples
Say I have a dataset of n samples. I want to maximize every feature’s value. I’m not sure if feature 1 is more important than feature 2, etc. Are there any methods of ranking my samples out there? If ...
- 1
0
votes
1
answer
34
views
Statistical test for comparing number of clusters in data
I am performing $K$-means clustering on a dataset consisting of $n$ observations and $d$ variables, and I'm trying to determine the optimal number of clusters. Is there a test that can determine the ...
- 121
2
votes
3
answers
761
views
What's the fastest clustering package in Python?
I'd like to perform clustering analysis on a dataset with 1,300 columns and 500,000 rows.
I've seen that clustering algorithms are available in SciKit-Learn. But I'm worried that the algorithms will ...
- 617
2
votes
1
answer
20
views
Im looking for good neurons silmilarity metric
Recently I managed to create simple neural network visualization, to help to understand how neural network works on the signal level. I also wanted to arrange neurons by similarity cause I was ...
- 23
0
votes
0
answers
18
views
Scaling nominal vars K means
I had a discussion recently with a coworker. We are running a K means clustering algorithm. He said that when dummy variables are made into 1s and 0s,these columns must be scaled in a specific way. If ...
- 1
1
vote
1
answer
312
views
Topic classification on text data with no/few labels
I would like to achieve a classification of a text input into predefined categories.
From what I have understand unsupervised approach are unfeasible if my target label is something very rare in ...
- 45
0
votes
1
answer
527
views
How to compare labels from clustering analysis and original ones?
I was asked to run a clustering analysis to assess the validity of labels for a manually labelled dataset.
I can simply save the actual labels (4 classes: 0, 1, 2, 3) and run clustering analysis (let'...
0
votes
0
answers
28
views
General approach for "regime detection" within timeseries data
Assume a stock market type of dataset with a handful of timeseries, representing volume of trades, volatility of trades, and similar meta metrics. If necessary, assume that is on an hourly basis.
What ...
- 101
0
votes
1
answer
31
views
ML Predicted Model for 2 values
I have a data set with 96 rows. It contains date, source, spend and number of customers.
I have 4 different sources that generate customers and you can see in the dataset how much I spend and how many ...
0
votes
2
answers
106
views
NLP topic clustering
In my dataset, I have 500 abstracts. The goal is to cluster them in 2 topics.
One topic must have those abstracts which contain some list of words or similar words and the rest of the abstracts in ...
1
vote
1
answer
20
views
Underfitting and perfomance metrics in unsupervised methods
My question is simple and yet quite hard to find an answer to. In an unsupervised method, for example, when you have to reconstruct an input, how can you tell if your loss is good enough? Generally, ...
- 11
0
votes
0
answers
10
views
Unsupervised vine trunk foreground segmentation
I'm currently working on a computer vision project for a vineyard robot. I trained a robust object detection for the vine trunks but now I need to apply semantic segmentation on the trunks so I can ...
1
vote
1
answer
30
views
Predicting many classes, is it a known solution to build n-group classifiers?
Imagine you want to predict 2048 classes. Instead of asking one model to predict all of them at once, is it a known type of solution to have a model predict which cluster or group of classes an input ...
1
vote
1
answer
46
views
Looking for an approach to automatically recognize faulted data
I'm currently looking into different approaches to handle the following problem:
Let's say there is an X amount of Class A which embodies some kind of absence of work on a daily basis. E.g. Vacation, ...
- 21
0
votes
0
answers
36
views
Video anomaly detection/ Evaluation AUC
I have trained an unsupervised anomaly detector for surveillance videos. After inference, I rescale the scores between max/min from the resulting scores array.
scores = (scores - min(scores))/max(...
- 1
0
votes
0
answers
19
views
How to find a unique feature vector for retail items?
I'm working on this problem where we get images of SKUs (Stock Keeping Unit) or in other words retail products and my job is to classify which product is it.
I want to find a unique feature vector ...
- 101
0
votes
1
answer
35
views
detecting abnormality in a specific feature with respect to others (unsupervised?)
I have a large dataset with a feature y which is dependent in part on features x1 and x2. All features are noisy, and y is also dependent on other parameters not captured in the dataset. I would like ...
1
vote
1
answer
179
views
Encode categorical data for unsupervised learning
What is the best encoder for categorical data in unsupervised learning?
I am using unsupervised learning on mixed data (such as K-means).
Before running my unsupervised algorithm, I am using dimension ...
- 11
0
votes
0
answers
15
views
Can/should a logical multi-class classification anomaly detection system be described as "unsupervised machine learning"?
I would like to ensure that my use of terminology is accurate. My question is: what terminology should I be using in this case?
The system I am building assigns classes (-1, 0, +1) to observations ...
0
votes
2
answers
323
views
Time Series - Anomaly Detection
I have time-series data with alerts (every minute) that I need to find anomalies in.
I am looking for a library which can do unsupervised learning of this data and detect anomalies in the data.
Which ...
- 101
1
vote
2
answers
61
views
How to cluster components of a graph containing text data?
Suppose that I have a graph that has components like the image below.
Graph nodes contain text data (titles) and the edges data is the similarity (percentage).
I know that each component represents a ...
2
votes
1
answer
434
views
In Orange Data Mining, how do I use results from clustering a training-set to test and score a test-set?
I am performing analysis on the well-known 'Adult' data-set, available on UCI using Orange Data Mining. In a PhD thesis, Pelleg (2004; pg 79) uses unsupervised clustering of the prescribed training ...
- 121
1
vote
1
answer
997
views
Clustering for Sparse Data Matrix of high dimension
I currently have a dataset of 1000 entries with 512 features that are sparse. I want to cluster them. I have attempted using kmeans, but found that the clustering wasn't very good, and have been ...
- 11
1
vote
2
answers
56
views
Validate Unsupervised Binary Classification
I’m working on a fully unsupervised anomaly detection problem. Since it’s completely unsupervised, I’m having hard times in defining some metrics to kind of validate the results (I run several ...
- 11
0
votes
0
answers
14
views
Find the most impactfuls parameters multivariate output unsupervised ML
I am currently on a proect where my df has more than 600 parameters of analog sensors (A parameters) and about 50 other parameters (F parameters). I want to find for each of these 50 parameters (F ...
- 1