Questions tagged [dimensionality-reduction]

Dimensionality reduction refers to techniques for reducing many variables into a smaller number while keeping as much information as possible. One prominent method is [tag pca]

Filter by
Sorted by
Tagged with
0 votes
0 answers
37 views

T-sne Inconsistent Results - Solvable?

The algorithm is highly dependent on hyperparameters such as perplexity and can yield dissimilar visualizations as a result. However, I was wandering if in the case of clear clusters in the high ...
codeplay's user avatar
1 vote
1 answer
48 views

Beginner basic clustering model and one-hot encoding?

I have a dataframe of natural disaster incidents in Afghanistan from 2016 - 2023. Column names: REGION (Northern, Eastern etc) PROV_CODE (province) PROV_NAME DIST_CODE (district) DIST_NAME INC_DATE (...
Mas's user avatar
  • 13
0 votes
1 answer
31 views

Beginner clustering project, what are the input features and how do I analyze the data?

I am a beginner to data science. I have this dataset on natural disaster events in Afghanistan from 2016 - 2017. Columns: REGION (ex. North, North West, etc) PROVINCE_NAME (kind of like US 50 states) ...
Mas's user avatar
  • 13
0 votes
0 answers
13 views

How to deal with a small extra cluster in a tabular data?

I am working on a high dimensional tabular dataset with 1600 features and 9440 rows. No matter how I select the features, when I try to project my data into a 2d or 3d graph using dimensionality ...
Tanmay Sharma's user avatar
0 votes
0 answers
12 views

Is there any advantage to providing multi-dimensional input to torch modules?

Most layer types in torch.nn such as torch.nn.Linear accept input with more than one dimension. Is there any advantage in doing so if you can shape your data to represent a certain arrangement in ...
kot's user avatar
  • 1
0 votes
1 answer
37 views

How to explain the new features after a PCA?

Let's say I made a PCA in which I reduced from 10 dimensions to 3. And it clusters the classes correctly, but how do I explain which dimensions are better to predict? It is obvious that the 3 ...
MAD MAGGOT's user avatar
0 votes
0 answers
10 views

What is the best sampling strategy for correlation analysis?

I have a big dataset, and i want to finds subspaces with high correlation among features. I want to take only samples of data. So, what is the best sampling strategy in this context. Thanks
Imen F's user avatar
  • 21
0 votes
0 answers
8 views

A query with regard to removing data from a dataset before clustering. - conceptual

I am in posession of data with regard to my domain which is energy economics. The dataset contains daily data on daily electricity demand along with the daily capacities of wind and solar plants for ...
user154329's user avatar
0 votes
1 answer
64 views

Linear Model With Highly Correlated Attributes Producing Inconsistent Weights

I know that having correlated attributes violates the linear model assumption of independent attributes, and I'm not interested in creating a more sophisticated model to tease apart the dependent ...
Brett L's user avatar
1 vote
0 answers
32 views

Density distribution for feature analysis

I trained a ML model on original data with 6373 features, then I trained the same model on compressed data (using autoencoder) and I got an improvement. Finally, I trained the same model on reduced ...
Skander Hamdi's user avatar
0 votes
0 answers
23 views

Using Autoencoder in Python for Data Selection and Risk Level Calculation

I have a dataset that includes 100 data points from each sensor, representing various measurements. These measurements can be used to calculate the level of risk associated with each sensor. However, ...
JACK1's user avatar
  • 1
0 votes
0 answers
11 views

Problem with 2 variable PCA loadings- Loadings are the same for all variables

I'm working on a problem for which i want to do some dimensionality reduction using 3 different PCAs of 2 variables each. Basically i want to perform a PCA and keep the first component between the ...
Felipe Maresca's user avatar
1 vote
0 answers
20 views

How to properly visualize high-dimensional embeddings along with the decision boundary in 2-D?

I have a number of embeddings (300-dimensional FastText vectors for each instance of each class) that I apply a classifier to (Logistic Regression for now). I want to visualize the embeddings as well ...
Metrician's user avatar
2 votes
1 answer
77 views

How to improve the preservation of the global data structure in UMAP?

I have a dataset, where the features are comprised of points arranged in a regular grid on a simplex. Each of these points are defined as follows: A point $\mathbf{x}$ on the simplex can be ...
pmu2022's user avatar
  • 121
0 votes
0 answers
21 views

Can TSNE and other visualisation methods separate multivariate normal blobs?

Consider we have two classes of points. Both of them come from a multivariate normal distribution with an unrestricted covariance matrix. Let's assume, that the densities of those distributions do not ...
Karol Szustakowski's user avatar
0 votes
0 answers
134 views

How to cluster multivariate time-series on different datasets?

I have following issue: I am trying to cluster similar countries with respect to different temporal features. Therefore, I have twelve different datasets each representing a different country. Each ...
sapphire's user avatar
0 votes
0 answers
187 views

Transformer-based autoencoder generates same output and bad embeddings

I am trying to implement the transformer-based autoencoder presented in this paper: https://arxiv.org/abs/2210.08288 The paper seems rather vague to me and I do not fully understand how the model is ...
Droidenkiller's user avatar
0 votes
1 answer
35 views

important feature selection using dimensionality reduction algorithms

I have a dataset having more than 25000 features. I did perform noise removal using the histogram approach, and this dataset gets reduced to more than 5000 features. There are two classes, healthy and ...
Gajanan Kothawade's user avatar
0 votes
0 answers
34 views

Feature selection in high-dimensional datasets with sparse features

What are the most effective techniques for feature selection in high-dimensional datasets with sparse features in the field of natural language processing?
Todd Takala's user avatar
0 votes
0 answers
28 views

Why use feature-hashing instead of just remove random words from Bag-of-words?

As far as I understand Feature-Hashing ("Hashing Trick") is that we map some string to an index in an array e.g say we want the resulting dimension of our array to be 5, then maybe the text <...
CutePoison's user avatar
0 votes
1 answer
226 views

How to efficiently reduce dimensions of one-hot encoded categorical values?

I'm currently working on a project where I'm using an LSTM to learn and predict sequences of categorical data. My dataset consists of variable-length sequences of items $s_i = [x_{i_0}, x_{i_1}, ..., ...
acxcv's user avatar
  • 13
-1 votes
1 answer
43 views

Determining "filters" dimension after a convolution operation

I tried to calculate the "filtered" dimension and I seem to be getting it wrong. Below there is the image I am trying to calculate the "filtered" dimension for, where you have 192 ...
Mah Neh's user avatar
0 votes
0 answers
13 views

Larger Latent Space Dim for Point Cloud Autoencoder

So I'm trying to follow a paper that uses a AE to learn point clouds. The thing is, the dimension of the point cloud data is 3 (x, y, z), but the dimension of the latent space from what I can tell is ...
okayatcp12's user avatar
0 votes
0 answers
14 views

Preserving the frontier while reducing dimensionality

I have this problem: reduce the dimensionality of the data such that points belonging to the frontier of the convex hull in high dimension stay relatively close the frontier in low dimensions. I do ...
Michael Bay's user avatar
0 votes
0 answers
17 views

How to handle dimensionality differences over time or between subjects

Note: This question has in mind tabular data, rather than imaging/NLP. In the situation of collecting data over long periods of time, instruments may change and collect more precise data. This leads ...
user3219765's user avatar
0 votes
0 answers
19 views

How to choose the optimal PCA kernel

In a chemometrics application, I need to reduce the dimensionality of a spectral scan. The standard PCA is linear. Not sure if the data is. How do I choose the most optimal PCA kernel?
machinelearner's user avatar
1 vote
1 answer
121 views

Clustering by using Locality sensitive hashing *after* Random projection

It is well known that Random Projection (RP) is tightly linked to Locality Sensitive Hashing (LSH). My goal is to cluster a large number of points lying in a d-dimensional Euclidean space, where $d$ ...
Penelope Benenati's user avatar
0 votes
0 answers
68 views

Is it possible to use text Auto Encoders without text generation?

I have a use case where I have large texts, and a lot of it. Pretty often the sequence length exceeds 1000 tokens. I need a lower dimensional compression of the texts as an input for a classifier. The ...
thijsvdp's user avatar
  • 101
0 votes
0 answers
15 views

Reducing the dimensionality of entire dataset

I have a lot of datasets with different shapes. For example few of them are (90, 892), (74, 853), (93, 765), ... etc. I want to convert this shape to (90, 4), (74, 4), (93, 4) ... (x, 4). And, after ...
Shikhar's user avatar
1 vote
1 answer
107 views

How to describe this UMAP connectivity figure

I generated this UMAP connectivity diagram of my research data. How do I interpret/describe this plot regarding the UMAP connectivity? Is it correct to say that: As there is a lot of connectivity ...
Joe's user avatar
  • 147
0 votes
1 answer
49 views

Classification problem with no context in numerical features

I have an extremely abstract and numeric data with equally abstract objective. I have around 3000 rows of train data (df_train), where I have a binary target ...
jstaxlin's user avatar
  • 118
2 votes
0 answers
74 views

How to understand to what maximum size you can reduce the dimension of data and avoid the curse of the dimensionality?

i have a question, maybe someone could help me. I use t-sne (also tried umap) to reduce the dimensionality of the text embeddings dataset (size of embedding 300). after that I will cluster using ...
Ir8_mind's user avatar
  • 183
0 votes
0 answers
27 views

How can "profiling" be done in a dataset? What are the different techniques?

I am currently doing an analysis in which I need to "profile" each record. For example, let's say I have a dataset of accounts with customer information (name, id, address, money spent, ...
thesadclown's user avatar
0 votes
0 answers
17 views

PCA, Better performances with 300 components rather than 400 components : why?

I am building this content based image retrieval system. I basically extract feature maps of size 1024x1x1 using any backbone. I then proceed to apply PCA on the extracted features in order to ...
Valentin Fontanger's user avatar
1 vote
1 answer
107 views

What is the Purpose of Feature Selection

I have a small medical dataset (200 samples) that contains only 6 cases of the condition I am trying to predict using machine learning. So far, the dataset is not proving useful for predicting the ...
sums22's user avatar
  • 417
0 votes
0 answers
39 views

How to Approach Linear Machine-Learning Model When Input Variables are Inconsistent

Disclaimer: I'm relatively new to the data science and ML world -- still trying to get a firm grasp on the fundamentals. I'm trying to overcome a regression challenge involving a large, multi-...
Austin Prater's user avatar
0 votes
1 answer
90 views

An autoencoder setup for anomaly detection

I am doing anomaly detection using machine learning. i have tried different models like isolation forest, SVM and KNN. The maximum accuracy that I can get from each of them is $80\%$ accordind to my ...
Riva11's user avatar
  • 11
-1 votes
2 answers
158 views

Why Do a Set of 3 Clusters Across 1 Dimension and a Set of 3 Clusters Across 2 Dimensions Form 9 Apparent Clusters in 3 Dimensions?

I am sorry if this is a well-known phenomenon but I can't quite wrap my head around this. I have a related question: How To Develop Cluster Models Where the Clusters Occur Along Subsets of Dimensions ...
from keras import michael's user avatar
1 vote
0 answers
37 views

Accuracy drops when adding a fully connected layer for dimensionality reduction to a ResNet50

I'm training a ResNet50 for image classification and I'm interested in decreasing the dimensionality of the embedded layer, in order to apply some clustering techniques. The suggested dimension is ...
Gello's user avatar
  • 61
0 votes
0 answers
21 views

Single scalar from vector

I am aware that this question is very general, but I found this question and it made me curious. What are the sensible ways that you can think of to derive a single scalar value from a vector? Of ...
a_gdevr's user avatar
  • 21
0 votes
1 answer
25 views

Dimensionality Reduction of Curved Structural Data

I have been using PCA dimensionality reduction on datasets that are quite linear and now I am tasked with the same on datasets that are largely curved in space. Imagine a noisy sine wave for ...
Bryon's user avatar
  • 111
1 vote
1 answer
22 views

How to choose Recursive Feature Elimination parameters

in my project I have >900 features and I thought to use Recursive Feature Elimination algorithm to reduce the dimensionality of my problem (in order to improve the accuracy). But I can't figure out ...
Giorgio Martinez's user avatar
0 votes
0 answers
30 views

Searching machine learning algorithm for regression problem with many features

I have a machine learning problem with about 160 features and 400 cases and I want to find the best predictors for a continuous outcome. The dataset contains variables of psychotherapists and clients. ...
Christopher Lalk's user avatar
2 votes
1 answer
1k views

What is the meaning of preserving local or global structure of the data?

I read about PaCMAP dimensionality reduction method (PaCMAP). They wrote that this method preserving both local and global structure of the data in original space. ...
Boom's user avatar
  • 297
1 vote
2 answers
173 views

Using PCA for Dimensionality Expansion

I was trying to use t-SNE algorithm for dimensionality reduction and I know this was not the primary usage of this algorithm and not recommended. I saw an implementation here. I am not convinced about ...
yeyosef's user avatar
  • 13
2 votes
1 answer
383 views

Can one use PCA to reduce the dimensionality of One-Hot-Encoded data?

I read a couple times that PCA was used as a method to reduce dimensionality for one-hot-encoded data. However, there were also some comments that using PCA is not a good idea since one-hot-encoded ...
2much2code's user avatar
1 vote
0 answers
19 views

Nearest neighbor face recognition in eigenspace when using dot product of test set with eigenvectors does not match the performance when using sklearn

I am trying to perform Face recognition using PCA (eigenfaces). I have a set of N training images (of dimensions M=wxh), which I have pre-processed into a vertical ...
zr0gravity7's user avatar
1 vote
0 answers
28 views

Understanding Linear Discriminant Analysis

I am trying to understand the results of a Linear Discriminant Analysis. For that purpose I used the Iris dataset. I plotted the two first features: Then, I projected these two features in the new ...
Victor Guijarro's user avatar
1 vote
2 answers
260 views

Dimensionality reduction for millions of features

I have a dataset with 10 million observations and 1 million sparse features. I would like to build a binary classifier for predicting a particular feature of interest. My main problem is how to deal ...
j k's user avatar
  • 11
2 votes
1 answer
2k views

Can t-SNE be applied to visualize time series datasets

I have multiple time-series datasets containing 9 IMU sensor features. Suppose I use the sliding window method to split all these data into samples with the sequence length of 100, i.e. the dimension ...
CuishleChen's user avatar

1
2 3 4 5 6