Questions tagged [unsupervised-learning]
Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.
451
questions
0
votes
0
answers
12
views
Ranking risky routes
i'm looking to get the top 10 rank of the most dangerous routes. I have a routes table where each row is a route and it has features such as
avg daily traffic for past three years
No. of times where ...
- 1
1
vote
1
answer
30
views
Removing specific phrases from textual data (R)
I have the following reddit posts and I would like to clean the posts and remove from the data the specific phrase "Click to expand", while keeping all other words within a post the same.
<...
- 127
0
votes
0
answers
12
views
Unsupervised rule extraction of categorical data
I have a dataset of network traffic with three features that I would like to extract rules from in order to apply firewall/flow control rules i.e. the permitted flows.
I am able to classify a ...
0
votes
1
answer
36
views
9-dimensional data classification
I have a dataset with 9 columns and 20 million rows. I have now split the dataset into two parts using IQR.
One part is my inliers, the other part is my outliers.
I now want to build a categorizer ...
1
vote
1
answer
51
views
How to Justify Anomalies Detected by Unsupervised Anomaly Detection Models? [closed]
I'm working on an unsupervised anomaly detection project involving a large sensor dataset, where I aim to identify anomalies without the aid of labeled data. While I've implemented several ...
0
votes
1
answer
28
views
Overfitting in the trained model
For my project on the classification problem of predicting churn customers, I trained various base models using k-fold validation on the training dataset and out of which random forest gave the best ...
0
votes
0
answers
11
views
How to update the weights of a ranking algorithms through unsupervised learning?
I have, let's say, 100 samples with n features, and I want to rank them by learning the weights for each feature. Weights must be learned from implicit ratings from the user (eg. user click). However, ...
0
votes
1
answer
28
views
Different scaling methods of different features results in a faux dependency between them
My dataset contains the following two features: "movie duration" (minutes) and "tv shows duration" (seasons). If a certain sample is of type "movie", it's duration will ...
- 1
0
votes
1
answer
52
views
Clustering task: drop or not drop a categorical attribute/feature for which each row in the dataset contains a different value
I am dealing with a clustering task. In the dataset I am using there is a categorical feature and for each row in the dataset I have a different value for that feature (my dataset consists of 1000 ...
- 1
0
votes
0
answers
10
views
What kind of learning do I need ? (use-case specific)
Consider a scenario where I have a model trained on gesture videos (say a 3D ResNet). I am looking for a technique (or a combination) that allows me to further train the model every time I have a new ...
- 139
0
votes
0
answers
15
views
Using Autoencoder in Python for Data Selection and Risk Level Calculation
I have a dataset that includes 100 data points from each sensor, representing various measurements. These measurements can be used to calculate the level of risk associated with each sensor. However, ...
- 1
0
votes
0
answers
25
views
Equivalence between K-Means objective functions proof
I am working on a project involving the K-Means clustering algorithm, and I am trying to prove the equivalence between different formulations of the objective function. Specifically, I want to show ...
0
votes
0
answers
12
views
Unable to reproduce result for CIFAR10 with ResNet50 backbone and SIMCLR self-supervised algorithm
I am recently working on self-supervised learning, particularly simclr paradigm. I found it hard to reproduce results on cifar10 with linear evaluation. I now get round 60%, while the paper reports 80%...
- 1
2
votes
1
answer
71
views
How to improve the preservation of the global data structure in UMAP?
I have a dataset, where the features are comprised of points arranged in a regular grid on a simplex. Each of these points are defined as follows: A point $\mathbf{x}$ on the simplex can be ...
- 121
0
votes
0
answers
21
views
Can TSNE and other visualisation methods separate multivariate normal blobs?
Consider we have two classes of points. Both of them come from a multivariate normal distribution with an unrestricted covariance matrix.
Let's assume, that the densities of those distributions do not ...
0
votes
1
answer
211
views
Rigth way to find Lorentzian distance between 2 point
Following this paper and this paper, I'm trying to implement the formula for the Lorentzian distance between 2 points (aka the distance between 2 points in Lorentzian space). I'll use this a the ...
1
vote
0
answers
15
views
What algorithm should I use when trying to find closest record matches when records contain both categorical and discrete attributes?
I have 100000 records that have discrete features like topic (analytics etc) and categorical features like ticket details (eg: I need help with analytics for my business). When creating a new record, ...
- 11
0
votes
0
answers
58
views
How to use IsolationForest for anomaly detection of a 1D-array, given given another feature?
[Beginner at ML]
Hi,
I would like to use unsupervised anomaly detection of spectrograms.
Currently, I am trying IsolationForest on a bunch of 1D-arrays (light intensity vs wavelength) in order to ...
- 1
0
votes
0
answers
23
views
Varying feature vector lengths for learning
[I am a total beginner in machine learning algorithms]
I have 10 spectrograms (lines) for phytoplankton (each composed of 288 points).
Each spectrogram is associated with a phytoplankton dendity data ...
- 1
0
votes
0
answers
21
views
Can sigmoid function applied on distance calculation of Self Organizing Map improve the accuracy?
I just thought how sigmoid function helps logistic regression making more accurate classification and wonder if it can be applied on distance calculation of Self Organizing Map and making more better/...
- 1
0
votes
0
answers
8
views
How do you know which examples of data are the most accurate classified using Self Organizing Map?
Here is an example of data where I apply the SOM general rule:
If distance1 > distance2 than class C1 else class C2 ; So in this example I use a self organizing map which clusters only two classes ...
- 1
0
votes
2
answers
862
views
Why is UMAP used in combination with other Clustering Algorithm?
I've noticed that UMAP is often used in combination with other clustering algorithms, such as K-means, DBSCAN, HDBSCAN. However, from what I've understood, UMAP can be used for clustering tasks. So ...
- 103
0
votes
1
answer
14
views
Unsupervised ranking of samples
Say I have a dataset of n samples. I want to maximize every feature’s value. I’m not sure if feature 1 is more important than feature 2, etc. Are there any methods of ranking my samples out there? If ...
- 1
0
votes
1
answer
28
views
Statistical test for comparing number of clusters in data
I am performing $K$-means clustering on a dataset consisting of $n$ observations and $d$ variables, and I'm trying to determine the optimal number of clusters. Is there a test that can determine the ...
- 121
2
votes
3
answers
500
views
What's the fastest clustering package in Python?
I'd like to perform clustering analysis on a dataset with 1,300 columns and 500,000 rows.
I've seen that clustering algorithms are available in SciKit-Learn. But I'm worried that the algorithms will ...
- 607
2
votes
1
answer
20
views
Im looking for good neurons silmilarity metric
Recently I managed to create simple neural network visualization, to help to understand how neural network works on the signal level. I also wanted to arrange neurons by similarity cause I was ...
- 23
0
votes
0
answers
18
views
Scaling nominal vars K means
I had a discussion recently with a coworker. We are running a K means clustering algorithm. He said that when dummy variables are made into 1s and 0s,these columns must be scaled in a specific way. If ...
- 1
1
vote
1
answer
126
views
Topic classification on text data with no/few labels
I would like to achieve a classification of a text input into predefined categories.
From what I have understand unsupervised approach are unfeasible if my target label is something very rare in ...
- 45
0
votes
1
answer
375
views
How to compare labels from clustering analysis and original ones?
I was asked to run a clustering analysis to assess the validity of labels for a manually labelled dataset.
I can simply save the actual labels (4 classes: 0, 1, 2, 3) and run clustering analysis (let'...
0
votes
0
answers
26
views
General approach for "regime detection" within timeseries data
Assume a stock market type of dataset with a handful of timeseries, representing volume of trades, volatility of trades, and similar meta metrics. If necessary, assume that is on an hourly basis.
What ...
- 101
0
votes
1
answer
31
views
ML Predicted Model for 2 values
I have a data set with 96 rows. It contains date, source, spend and number of customers.
I have 4 different sources that generate customers and you can see in the dataset how much I spend and how many ...
0
votes
2
answers
83
views
NLP topic clustering
In my dataset, I have 500 abstracts. The goal is to cluster them in 2 topics.
One topic must have those abstracts which contain some list of words or similar words and the rest of the abstracts in ...
1
vote
1
answer
20
views
Underfitting and perfomance metrics in unsupervised methods
My question is simple and yet quite hard to find an answer to. In an unsupervised method, for example, when you have to reconstruct an input, how can you tell if your loss is good enough? Generally, ...
- 11
0
votes
0
answers
8
views
Unsupervised vine trunk foreground segmentation
I'm currently working on a computer vision project for a vineyard robot. I trained a robust object detection for the vine trunks but now I need to apply semantic segmentation on the trunks so I can ...
1
vote
1
answer
30
views
Predicting many classes, is it a known solution to build n-group classifiers?
Imagine you want to predict 2048 classes. Instead of asking one model to predict all of them at once, is it a known type of solution to have a model predict which cluster or group of classes an input ...
1
vote
1
answer
46
views
Looking for an approach to automatically recognize faulted data
I'm currently looking into different approaches to handle the following problem:
Let's say there is an X amount of Class A which embodies some kind of absence of work on a daily basis. E.g. Vacation, ...
- 21
0
votes
0
answers
31
views
Video anomaly detection/ Evaluation AUC
I have trained an unsupervised anomaly detector for surveillance videos. After inference, I rescale the scores between max/min from the resulting scores array.
scores = (scores - min(scores))/max(...
- 1
0
votes
0
answers
19
views
How to find a unique feature vector for retail items?
I'm working on this problem where we get images of SKUs (Stock Keeping Unit) or in other words retail products and my job is to classify which product is it.
I want to find a unique feature vector ...
- 101
0
votes
1
answer
33
views
detecting abnormality in a specific feature with respect to others (unsupervised?)
I have a large dataset with a feature y which is dependent in part on features x1 and x2. All features are noisy, and y is also dependent on other parameters not captured in the dataset. I would like ...
1
vote
1
answer
127
views
Encode categorical data for unsupervised learning
What is the best encoder for categorical data in unsupervised learning?
I am using unsupervised learning on mixed data (such as K-means).
Before running my unsupervised algorithm, I am using dimension ...
- 11
0
votes
0
answers
170
views
Custom *metric* parameter for distance matrix in unsupervised clustering methods
This is list of all the methods in unsupervised clustering methods:
https://scikit-learn.org/stable/modules/clustering.html
and out of these methods, following methods do not take custom pairwise ...
- 9
0
votes
0
answers
13
views
Can/should a logical multi-class classification anomaly detection system be described as "unsupervised machine learning"?
I would like to ensure that my use of terminology is accurate. My question is: what terminology should I be using in this case?
The system I am building assigns classes (-1, 0, +1) to observations ...
0
votes
2
answers
257
views
Time Series - Anomaly Detection
I have time-series data with alerts (every minute) that I need to find anomalies in.
I am looking for a library which can do unsupervised learning of this data and detect anomalies in the data.
Which ...
- 101
0
votes
0
answers
107
views
How to validate an Isolation Forest for unsupervised outlier detection
Im currently building a binary classification model for outlier detection using isolation forest. I predicted some outliers and now i should validate my model.
Im most unsupervised examples that ive ...
- 23
1
vote
2
answers
51
views
How to cluster components of a graph containing text data?
Suppose that I have a graph that has components like the image below.
Graph nodes contain text data (titles) and the edges data is the similarity (percentage).
I know that each component represents a ...
0
votes
0
answers
19
views
(anomaly detection) Is it wise to include human annotated y' in anomaly detection in order to detect the true outlier y?
I am doing a fraud detection project. Right now, I have feature variables and binary dependent variable y' which is labelled using heuristics. Ideally, I need to automate this manual process and ...
2
votes
1
answer
324
views
In Orange Data Mining, how do I use results from clustering a training-set to test and score a test-set?
I am performing analysis on the well-known 'Adult' data-set, available on UCI using Orange Data Mining. In a PhD thesis, Pelleg (2004; pg 79) uses unsupervised clustering of the prescribed training ...
- 121
0
votes
0
answers
30
views
A/B testing for channel preference
Right now my organization runs a lot of promotional campaign every month. We send email, SMS and WhatsApp to all customers for each campaign. I am running a project to identity the best channel for a ...
1
vote
1
answer
830
views
Clustering for Sparse Data Matrix of high dimension
I currently have a dataset of 1000 entries with 512 features that are sparse. I want to cluster them. I have attempted using kmeans, but found that the clustering wasn't very good, and have been ...
- 11
1
vote
2
answers
50
views
Validate Unsupervised Binary Classification
I’m working on a fully unsupervised anomaly detection problem. Since it’s completely unsupervised, I’m having hard times in defining some metrics to kind of validate the results (I run several ...
- 11