Questions tagged [unsupervised-learning]

Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.

Filter by
Sorted by
Tagged with
1 vote
1 answer
30 views

Removing specific phrases from textual data (R)

I have the following reddit posts and I would like to clean the posts and remove from the data the specific phrase "Click to expand", while keeping all other words within a post the same. <...
maldini1990's user avatar
0 votes
0 answers
11 views

Unsupervised rule extraction of categorical data

I have a dataset of network traffic with three features that I would like to extract rules from in order to apply firewall/flow control rules i.e. the permitted flows. I am able to classify a ...
user153170's user avatar
0 votes
1 answer
35 views

9-dimensional data classification

I have a dataset with 9 columns and 20 million rows. I have now split the dataset into two parts using IQR. One part is my inliers, the other part is my outliers. I now want to build a categorizer ...
Sponge_Data_45's user avatar
1 vote
1 answer
48 views

How to Justify Anomalies Detected by Unsupervised Anomaly Detection Models? [closed]

I'm working on an unsupervised anomaly detection project involving a large sensor dataset, where I aim to identify anomalies without the aid of labeled data. While I've implemented several ...
Jais Varghese Joseph's user avatar
0 votes
1 answer
28 views

Overfitting in the trained model

For my project on the classification problem of predicting churn customers, I trained various base models using k-fold validation on the training dataset and out of which random forest gave the best ...
Yuvika Yadav's user avatar
0 votes
0 answers
11 views

How to update the weights of a ranking algorithms through unsupervised learning?

I have, let's say, 100 samples with n features, and I want to rank them by learning the weights for each feature. Weights must be learned from implicit ratings from the user (eg. user click). However, ...
Naman Lazarus's user avatar
0 votes
1 answer
27 views

Different scaling methods of different features results in a faux dependency between them

My dataset contains the following two features: "movie duration" (minutes) and "tv shows duration" (seasons). If a certain sample is of type "movie", it's duration will ...
Shir's user avatar
  • 1
0 votes
1 answer
44 views

Clustering task: drop or not drop a categorical attribute/feature for which each row in the dataset contains a different value

I am dealing with a clustering task. In the dataset I am using there is a categorical feature and for each row in the dataset I have a different value for that feature (my dataset consists of 1000 ...
vito97's user avatar
  • 1
0 votes
0 answers
10 views

What kind of learning do I need ? (use-case specific)

Consider a scenario where I have a model trained on gesture videos (say a 3D ResNet). I am looking for a technique (or a combination) that allows me to further train the model every time I have a new ...
batman's user avatar
  • 139
0 votes
0 answers
15 views

Using Autoencoder in Python for Data Selection and Risk Level Calculation

I have a dataset that includes 100 data points from each sensor, representing various measurements. These measurements can be used to calculate the level of risk associated with each sensor. However, ...
JACK1's user avatar
  • 1
0 votes
0 answers
25 views

Equivalence between K-Means objective functions proof

I am working on a project involving the K-Means clustering algorithm, and I am trying to prove the equivalence between different formulations of the objective function. Specifically, I want to show ...
Oren Ben Eliyahu's user avatar
0 votes
0 answers
12 views

Unable to reproduce result for CIFAR10 with ResNet50 backbone and SIMCLR self-supervised algorithm

I am recently working on self-supervised learning, particularly simclr paradigm. I found it hard to reproduce results on cifar10 with linear evaluation. I now get round 60%, while the paper reports 80%...
noah0822's user avatar
2 votes
1 answer
70 views

How to improve the preservation of the global data structure in UMAP?

I have a dataset, where the features are comprised of points arranged in a regular grid on a simplex. Each of these points are defined as follows: A point $\mathbf{x}$ on the simplex can be ...
pmu2022's user avatar
  • 121
0 votes
0 answers
19 views

Can TSNE and other visualisation methods separate multivariate normal blobs?

Consider we have two classes of points. Both of them come from a multivariate normal distribution with an unrestricted covariance matrix. Let's assume, that the densities of those distributions do not ...
Karol Szustakowski's user avatar
0 votes
1 answer
186 views

Rigth way to find Lorentzian distance between 2 point

Following this paper and this paper, I'm trying to implement the formula for the Lorentzian distance between 2 points (aka the distance between 2 points in Lorentzian space). I'll use this a the ...
Ricardo Chavarria's user avatar
1 vote
0 answers
15 views

What algorithm should I use when trying to find closest record matches when records contain both categorical and discrete attributes?

I have 100000 records that have discrete features like topic (analytics etc) and categorical features like ticket details (eg: I need help with analytics for my business). When creating a new record, ...
Jack Smith's user avatar
0 votes
0 answers
57 views

How to use IsolationForest for anomaly detection of a 1D-array, given given another feature?

[Beginner at ML] Hi, I would like to use unsupervised anomaly detection of spectrograms. Currently, I am trying IsolationForest on a bunch of 1D-arrays (light intensity vs wavelength) in order to ...
Marianne's user avatar
0 votes
0 answers
23 views

Varying feature vector lengths for learning

[I am a total beginner in machine learning algorithms] I have 10 spectrograms (lines) for phytoplankton (each composed of 288 points). Each spectrogram is associated with a phytoplankton dendity data ...
Marianne's user avatar
0 votes
0 answers
20 views

Can sigmoid function applied on distance calculation of Self Organizing Map improve the accuracy?

I just thought how sigmoid function helps logistic regression making more accurate classification and wonder if it can be applied on distance calculation of Self Organizing Map and making more better/...
darieem's user avatar
0 votes
0 answers
8 views

How do you know which examples of data are the most accurate classified using Self Organizing Map?

Here is an example of data where I apply the SOM general rule: If distance1 > distance2 than class C1 else class C2 ; So in this example I use a self organizing map which clusters only two classes ...
darieem's user avatar
0 votes
2 answers
772 views

Why is UMAP used in combination with other Clustering Algorithm?

I've noticed that UMAP is often used in combination with other clustering algorithms, such as K-means, DBSCAN, HDBSCAN. However, from what I've understood, UMAP can be used for clustering tasks. So ...
coelidonum's user avatar
0 votes
1 answer
12 views

Unsupervised ranking of samples

Say I have a dataset of n samples. I want to maximize every feature’s value. I’m not sure if feature 1 is more important than feature 2, etc. Are there any methods of ranking my samples out there? If ...
mjw467's user avatar
  • 1
0 votes
1 answer
28 views

Statistical test for comparing number of clusters in data

I am performing $K$-means clustering on a dataset consisting of $n$ observations and $d$ variables, and I'm trying to determine the optimal number of clusters. Is there a test that can determine the ...
RyRy the Fly Guy's user avatar
2 votes
3 answers
464 views

What's the fastest clustering package in Python?

I'd like to perform clustering analysis on a dataset with 1,300 columns and 500,000 rows. I've seen that clustering algorithms are available in SciKit-Learn. But I'm worried that the algorithms will ...
Connor's user avatar
  • 607
2 votes
1 answer
20 views

Im looking for good neurons silmilarity metric

Recently I managed to create simple neural network visualization, to help to understand how neural network works on the signal level. I also wanted to arrange neurons by similarity cause I was ...
dismedia's user avatar
0 votes
0 answers
18 views

Scaling nominal vars K means

I had a discussion recently with a coworker. We are running a K means clustering algorithm. He said that when dummy variables are made into 1s and 0s,these columns must be scaled in a specific way. If ...
NKI's user avatar
  • 1
1 vote
1 answer
105 views

Topic classification on text data with no/few labels

I would like to achieve a classification of a text input into predefined categories. From what I have understand unsupervised approach are unfeasible if my target label is something very rare in ...
Andrea's user avatar
  • 45
0 votes
1 answer
342 views

How to compare labels from clustering analysis and original ones?

I was asked to run a clustering analysis to assess the validity of labels for a manually labelled dataset. I can simply save the actual labels (4 classes: 0, 1, 2, 3) and run clustering analysis (let'...
AngelMarcos's user avatar
0 votes
0 answers
25 views

General approach for "regime detection" within timeseries data

Assume a stock market type of dataset with a handful of timeseries, representing volume of trades, volatility of trades, and similar meta metrics. If necessary, assume that is on an hourly basis. What ...
Nik's user avatar
  • 101
0 votes
1 answer
31 views

ML Predicted Model for 2 values

I have a data set with 96 rows. It contains date, source, spend and number of customers. I have 4 different sources that generate customers and you can see in the dataset how much I spend and how many ...
Leonardo Urbiola's user avatar
0 votes
2 answers
74 views

NLP topic clustering

In my dataset, I have 500 abstracts. The goal is to cluster them in 2 topics. One topic must have those abstracts which contain some list of words or similar words and the rest of the abstracts in ...
Nency Bansal's user avatar
1 vote
1 answer
20 views

Underfitting and perfomance metrics in unsupervised methods

My question is simple and yet quite hard to find an answer to. In an unsupervised method, for example, when you have to reconstruct an input, how can you tell if your loss is good enough? Generally, ...
BilboBuggins's user avatar
0 votes
0 answers
8 views

Unsupervised vine trunk foreground segmentation

I'm currently working on a computer vision project for a vineyard robot. I trained a robust object detection for the vine trunks but now I need to apply semantic segmentation on the trunks so I can ...
hellvetica's user avatar
1 vote
1 answer
30 views

Predicting many classes, is it a known solution to build n-group classifiers?

Imagine you want to predict 2048 classes. Instead of asking one model to predict all of them at once, is it a known type of solution to have a model predict which cluster or group of classes an input ...
Jonathan Allen Grant's user avatar
1 vote
1 answer
46 views

Looking for an approach to automatically recognize faulted data

I'm currently looking into different approaches to handle the following problem: Let's say there is an X amount of Class A which embodies some kind of absence of work on a daily basis. E.g. Vacation, ...
bitfish31's user avatar
0 votes
0 answers
31 views

Video anomaly detection/ Evaluation AUC

I have trained an unsupervised anomaly detector for surveillance videos. After inference, I rescale the scores between max/min from the resulting scores array. scores = (scores - min(scores))/max(...
TecK97's user avatar
  • 1
0 votes
0 answers
19 views

How to find a unique feature vector for retail items?

I'm working on this problem where we get images of SKUs (Stock Keeping Unit) or in other words retail products and my job is to classify which product is it. I want to find a unique feature vector ...
Hissaan Ali's user avatar
0 votes
1 answer
33 views

detecting abnormality in a specific feature with respect to others (unsupervised?)

I have a large dataset with a feature y which is dependent in part on features x1 and x2. All features are noisy, and y is also dependent on other parameters not captured in the dataset. I would like ...
user18236139's user avatar
1 vote
1 answer
120 views

Encode categorical data for unsupervised learning

What is the best encoder for categorical data in unsupervised learning? I am using unsupervised learning on mixed data (such as K-means). Before running my unsupervised algorithm, I am using dimension ...
Julien PETOT's user avatar
0 votes
0 answers
165 views

Custom *metric* parameter for distance matrix in unsupervised clustering methods

This is list of all the methods in unsupervised clustering methods: https://scikit-learn.org/stable/modules/clustering.html and out of these methods, following methods do not take custom pairwise ...
Ziprip's user avatar
  • 9
0 votes
0 answers
11 views

Can/should a logical multi-class classification anomaly detection system be described as "unsupervised machine learning"?

I would like to ensure that my use of terminology is accurate. My question is: what terminology should I be using in this case? The system I am building assigns classes (-1, 0, +1) to observations ...
HackJob99's user avatar
0 votes
2 answers
250 views

Time Series - Anomaly Detection

I have time-series data with alerts (every minute) that I need to find anomalies in. I am looking for a library which can do unsupervised learning of this data and detect anomalies in the data. Which ...
Jilaba Hindga's user avatar
0 votes
0 answers
106 views

How to validate an Isolation Forest for unsupervised outlier detection

Im currently building a binary classification model for outlier detection using isolation forest. I predicted some outliers and now i should validate my model. Im most unsupervised examples that ive ...
arooki's user avatar
  • 23
1 vote
2 answers
50 views

How to cluster components of a graph containing text data?

Suppose that I have a graph that has components like the image below. Graph nodes contain text data (titles) and the edges data is the similarity (percentage). I know that each component represents a ...
Reza Shabrang's user avatar
0 votes
0 answers
19 views

(anomaly detection) Is it wise to include human annotated y' in anomaly detection in order to detect the true outlier y?

I am doing a fraud detection project. Right now, I have feature variables and binary dependent variable y' which is labelled using heuristics. Ideally, I need to automate this manual process and ...
Guangyi Chen's user avatar
2 votes
1 answer
308 views

In Orange Data Mining, how do I use results from clustering a training-set to test and score a test-set?

I am performing analysis on the well-known 'Adult' data-set, available on UCI using Orange Data Mining. In a PhD thesis, Pelleg (2004; pg 79) uses unsupervised clustering of the prescribed training ...
Paul Higgins's user avatar
0 votes
0 answers
30 views

A/B testing for channel preference

Right now my organization runs a lot of promotional campaign every month. We send email, SMS and WhatsApp to all customers for each campaign. I am running a project to identity the best channel for a ...
Subhashree subhadarsani's user avatar
0 votes
0 answers
18 views

Movement Analysis (Unsupervised Learning)

I have some data on the movement of a drone, D piloted by multiple people, P. Multiple metrics on the its motion is recorded, example speed, acceleration, elevation, angles which will be denoted by X1,...
mehmat's user avatar
  • 33
0 votes
0 answers
24 views

Adjusting the parameters for Isomap and Spectral Embedding/Eigenmap in sklearn?

I read around the basic ideas of these reduction methods, but I have no idea on how I would adjust the parameters for these so that it still reflects the original data as accuratley as possible. What ...
Is land's user avatar
  • 11
0 votes
0 answers
21 views

Issue with distance on kmedoids class python

I found a code for the kmedoids class that I wanted to use. I changed the distance to cosine. So I'm using a data matrix with 100 dimensions and roughly 150 data entries, and trying to get 2 clusters ...
Is land's user avatar
  • 11

1
2 3 4 5
10