Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [dimensionality-reduction]

Dimensionality reduction refers to techniques for reducing many variables into a smaller number while keeping as much information as possible. One prominent method is [tag pca]

Filter by
Sorted by
Tagged with
0 votes
1 answer
22 views

The latest approach for feature dimenesion reduction

I have a feature matrix with 1200 rows and 18930 columns. The matrix is sparse and the original paper has used a stacked denoising autoencoder for dimensionality reduction. Since I want to enhance the ...
Satarnejad's user avatar
1 vote
0 answers
73 views

Using UMAP on text data (euclidean distance on jaccard distance matrix)

I am checking the capabilities of the UMAP dimensionality reduction algorithm, I am not sure whether the approach I am using is valid and does not violate the rules/limitations of this algorithm. ...
rkabuk's user avatar
  • 21
0 votes
0 answers
24 views

Dimenson reduction from a cosine similarity matrix

I have a silly little question: I have 200 press articles (string), I vectorize these articles with an embedding model (sentence embedding), so I have 1024 values per article. I then have a 200 x 1024 ...
Bertrand's user avatar
0 votes
1 answer
30 views

TSNE plots of random data subsets are vastly different but labels are still clearly separated - what conclusions can we draw about the dataset?

I scraped a dataset of match data in a video game and labeled them according to their outcome (0 for loss, 1 for win). I wanted to see if there was actually any inherent relationship between the ...
Lilian Shi's user avatar
1 vote
1 answer
67 views

Beginner basic clustering model and one-hot encoding?

I have a dataframe of natural disaster incidents in Afghanistan from 2016 - 2023. Column names: REGION (Northern, Eastern etc) PROV_CODE (province) PROV_NAME DIST_CODE (district) DIST_NAME INC_DATE (...
Mas's user avatar
  • 55
0 votes
1 answer
49 views

Beginner clustering project, what are the input features and how do I analyze the data?

I am a beginner to data science. I have this dataset on natural disaster events in Afghanistan from 2016 - 2017. Columns: REGION (ex. North, North West, etc) PROVINCE_NAME (kind of like US 50 states) ...
Mas's user avatar
  • 55
0 votes
0 answers
15 views

How to deal with a small extra cluster in a tabular data?

I am working on a high dimensional tabular dataset with 1600 features and 9440 rows. No matter how I select the features, when I try to project my data into a 2d or 3d graph using dimensionality ...
Tanmay Sharma's user avatar
0 votes
0 answers
16 views

Is there any advantage to providing multi-dimensional input to torch modules?

Most layer types in torch.nn such as torch.nn.Linear accept input with more than one dimension. Is there any advantage in doing so if you can shape your data to represent a certain arrangement in ...
kot's user avatar
  • 11
0 votes
1 answer
98 views

How to explain the new features after a PCA?

Let's say I made a PCA in which I reduced from 10 dimensions to 3. And it clusters the classes correctly, but how do I explain which dimensions are better to predict? It is obvious that the 3 ...
MAD MAGGOT's user avatar
0 votes
0 answers
10 views

What is the best sampling strategy for correlation analysis?

I have a big dataset, and i want to finds subspaces with high correlation among features. I want to take only samples of data. So, what is the best sampling strategy in this context. Thanks
Imen F's user avatar
  • 21
0 votes
0 answers
8 views

A query with regard to removing data from a dataset before clustering. - conceptual

I am in posession of data with regard to my domain which is energy economics. The dataset contains daily data on daily electricity demand along with the daily capacities of wind and solar plants for ...
user154329's user avatar
0 votes
1 answer
81 views

Linear Model With Highly Correlated Attributes Producing Inconsistent Weights

I know that having correlated attributes violates the linear model assumption of independent attributes, and I'm not interested in creating a more sophisticated model to tease apart the dependent ...
Brett L's user avatar
1 vote
0 answers
33 views

Density distribution for feature analysis

I trained a ML model on original data with 6373 features, then I trained the same model on compressed data (using autoencoder) and I got an improvement. Finally, I trained the same model on reduced ...
Skander Hamdi's user avatar
0 votes
0 answers
28 views

Using Autoencoder in Python for Data Selection and Risk Level Calculation

I have a dataset that includes 100 data points from each sensor, representing various measurements. These measurements can be used to calculate the level of risk associated with each sensor. However, ...
JACK1's user avatar
  • 1
0 votes
0 answers
11 views

Problem with 2 variable PCA loadings- Loadings are the same for all variables

I'm working on a problem for which i want to do some dimensionality reduction using 3 different PCAs of 2 variables each. Basically i want to perform a PCA and keep the first component between the ...
Felipe Maresca's user avatar
1 vote
0 answers
22 views

How to properly visualize high-dimensional embeddings along with the decision boundary in 2-D?

I have a number of embeddings (300-dimensional FastText vectors for each instance of each class) that I apply a classifier to (Logistic Regression for now). I want to visualize the embeddings as well ...
Metrician's user avatar
2 votes
1 answer
96 views

How to improve the preservation of the global data structure in UMAP?

I have a dataset, where the features are comprised of points arranged in a regular grid on a simplex. Each of these points are defined as follows: A point $\mathbf{x}$ on the simplex can be ...
pmu2022's user avatar
  • 121
0 votes
0 answers
28 views

Can TSNE and other visualisation methods separate multivariate normal blobs?

Consider we have two classes of points. Both of them come from a multivariate normal distribution with an unrestricted covariance matrix. Let's assume, that the densities of those distributions do not ...
Karol Szustakowski's user avatar
0 votes
1 answer
42 views

important feature selection using dimensionality reduction algorithms

I have a dataset having more than 25000 features. I did perform noise removal using the histogram approach, and this dataset gets reduced to more than 5000 features. There are two classes, healthy and ...
Gajanan Kothawade's user avatar
0 votes
1 answer
422 views

How to efficiently reduce dimensions of one-hot encoded categorical values?

I'm currently working on a project where I'm using an LSTM to learn and predict sequences of categorical data. My dataset consists of variable-length sequences of items $s_i = [x_{i_0}, x_{i_1}, ..., ...
acxcv's user avatar
  • 13
-1 votes
1 answer
45 views

Determining "filters" dimension after a convolution operation

I tried to calculate the "filtered" dimension and I seem to be getting it wrong. Below there is the image I am trying to calculate the "filtered" dimension for, where you have 192 ...
Mah Neh's user avatar
1 vote
1 answer
142 views

Clustering by using Locality sensitive hashing *after* Random projection

It is well known that Random Projection (RP) is tightly linked to Locality Sensitive Hashing (LSH). My goal is to cluster a large number of points lying in a d-dimensional Euclidean space, where $d$ ...
Penelope Benenati's user avatar
0 votes
0 answers
74 views

Is it possible to use text Auto Encoders without text generation?

I have a use case where I have large texts, and a lot of it. Pretty often the sequence length exceeds 1000 tokens. I need a lower dimensional compression of the texts as an input for a classifier. The ...
thijsvdp's user avatar
  • 101
0 votes
0 answers
16 views

Reducing the dimensionality of entire dataset

I have a lot of datasets with different shapes. For example few of them are (90, 892), (74, 853), (93, 765), ... etc. I want to convert this shape to (90, 4), (74, 4), (93, 4) ... (x, 4). And, after ...
Shikhar's user avatar
1 vote
1 answer
124 views

How to describe this UMAP connectivity figure

I generated this UMAP connectivity diagram of my research data. How do I interpret/describe this plot regarding the UMAP connectivity? Is it correct to say that: As there is a lot of connectivity ...
Joe's user avatar
  • 147
0 votes
1 answer
52 views

Classification problem with no context in numerical features

I have an extremely abstract and numeric data with equally abstract objective. I have around 3000 rows of train data (df_train), where I have a binary target ...
jstaxlin's user avatar
  • 118
2 votes
0 answers
80 views

How to understand to what maximum size you can reduce the dimension of data and avoid the curse of the dimensionality?

i have a question, maybe someone could help me. I use t-sne (also tried umap) to reduce the dimensionality of the text embeddings dataset (size of embedding 300). after that I will cluster using ...
Ir8_mind's user avatar
  • 183
0 votes
0 answers
27 views

How can "profiling" be done in a dataset? What are the different techniques?

I am currently doing an analysis in which I need to "profile" each record. For example, let's say I have a dataset of accounts with customer information (name, id, address, money spent, ...
thesadclown's user avatar
0 votes
0 answers
19 views

PCA, Better performances with 300 components rather than 400 components : why?

I am building this content based image retrieval system. I basically extract feature maps of size 1024x1x1 using any backbone. I then proceed to apply PCA on the extracted features in order to ...
Valentin Fontanger's user avatar
1 vote
1 answer
132 views

What is the Purpose of Feature Selection

I have a small medical dataset (200 samples) that contains only 6 cases of the condition I am trying to predict using machine learning. So far, the dataset is not proving useful for predicting the ...
sums22's user avatar
  • 427
0 votes
0 answers
40 views

How to Approach Linear Machine-Learning Model When Input Variables are Inconsistent

Disclaimer: I'm relatively new to the data science and ML world -- still trying to get a firm grasp on the fundamentals. I'm trying to overcome a regression challenge involving a large, multi-...
Austin Prater's user avatar
0 votes
1 answer
127 views

An autoencoder setup for anomaly detection

I am doing anomaly detection using machine learning. i have tried different models like isolation forest, SVM and KNN. The maximum accuracy that I can get from each of them is $80\%$ accordind to my ...
Riva11's user avatar
  • 11
-1 votes
2 answers
165 views

Why Do a Set of 3 Clusters Across 1 Dimension and a Set of 3 Clusters Across 2 Dimensions Form 9 Apparent Clusters in 3 Dimensions?

I am sorry if this is a well-known phenomenon but I can't quite wrap my head around this. I have a related question: How To Develop Cluster Models Where the Clusters Occur Along Subsets of Dimensions ...
from keras import michael's user avatar
1 vote
0 answers
37 views

Accuracy drops when adding a fully connected layer for dimensionality reduction to a ResNet50

I'm training a ResNet50 for image classification and I'm interested in decreasing the dimensionality of the embedded layer, in order to apply some clustering techniques. The suggested dimension is ...
Gello's user avatar
  • 61
0 votes
0 answers
21 views

Single scalar from vector

I am aware that this question is very general, but I found this question and it made me curious. What are the sensible ways that you can think of to derive a single scalar value from a vector? Of ...
a_gdevr's user avatar
  • 21
0 votes
1 answer
28 views

Dimensionality Reduction of Curved Structural Data

I have been using PCA dimensionality reduction on datasets that are quite linear and now I am tasked with the same on datasets that are largely curved in space. Imagine a noisy sine wave for ...
Bryon's user avatar
  • 111
1 vote
1 answer
26 views

How to choose Recursive Feature Elimination parameters

in my project I have >900 features and I thought to use Recursive Feature Elimination algorithm to reduce the dimensionality of my problem (in order to improve the accuracy). But I can't figure out ...
Giorgio Martinez's user avatar
0 votes
0 answers
30 views

Searching machine learning algorithm for regression problem with many features

I have a machine learning problem with about 160 features and 400 cases and I want to find the best predictors for a continuous outcome. The dataset contains variables of psychotherapists and clients. ...
Christopher Lalk's user avatar
2 votes
1 answer
2k views

What is the meaning of preserving local or global structure of the data?

I read about PaCMAP dimensionality reduction method (PaCMAP). They wrote that this method preserving both local and global structure of the data in original space. ...
Boom's user avatar
  • 307
1 vote
2 answers
214 views

Using PCA for Dimensionality Expansion

I was trying to use t-SNE algorithm for dimensionality reduction and I know this was not the primary usage of this algorithm and not recommended. I saw an implementation here. I am not convinced about ...
yeyosef's user avatar
  • 13
2 votes
1 answer
749 views

Can one use PCA to reduce the dimensionality of One-Hot-Encoded data?

I read a couple times that PCA was used as a method to reduce dimensionality for one-hot-encoded data. However, there were also some comments that using PCA is not a good idea since one-hot-encoded ...
2much2code's user avatar
1 vote
0 answers
19 views

Nearest neighbor face recognition in eigenspace when using dot product of test set with eigenvectors does not match the performance when using sklearn

I am trying to perform Face recognition using PCA (eigenfaces). I have a set of N training images (of dimensions M=wxh), which I have pre-processed into a vertical ...
zr0gravity7's user avatar
1 vote
0 answers
33 views

Understanding Linear Discriminant Analysis

I am trying to understand the results of a Linear Discriminant Analysis. For that purpose I used the Iris dataset. I plotted the two first features: Then, I projected these two features in the new ...
Victor Guijarro's user avatar
1 vote
2 answers
327 views

Dimensionality reduction for millions of features

I have a dataset with 10 million observations and 1 million sparse features. I would like to build a binary classifier for predicting a particular feature of interest. My main problem is how to deal ...
j k's user avatar
  • 11
2 votes
1 answer
3k views

Can t-SNE be applied to visualize time series datasets

I have multiple time-series datasets containing 9 IMU sensor features. Suppose I use the sliding window method to split all these data into samples with the sequence length of 100, i.e. the dimension ...
CuishleChen's user avatar
0 votes
0 answers
24 views

Projection of PCA into new variable space (study area)

I have a set of >30 variables distributed in x,y coordinates across two study areas. I'd like to reduce that dimensionality with a PCA applied in the first study area. Then I'd like to visualize ...
Ricardo Guerreiro's user avatar
0 votes
0 answers
158 views

PCA vs t-SNE in asset pricing

So I am trying dimensionality reduction techniques on the S&P500 FY2020 data. I understand the CAPM model and the fact that doing a PCA determines my market variability factor (the first PCA ...
Soumik Mukherjee's user avatar
1 vote
1 answer
540 views

Factor Analysis vs PCA

Could someone please explain when FA is used or when PCA is used, as I understood FA do dimensionality reduction, however PCA - the main goal is the same. Then which one should I use and in which ...
Zexxxx's user avatar
  • 33
1 vote
1 answer
816 views

Reusing UMAP transformation

I have this use case: I want to apply a dimension reduction with UMAP to a initial dataset of high-dimension vectors (100d), and later, in second place, have the oppurtinity to add new data points ...
Nick9214's user avatar
4 votes
1 answer
3k views

What are the differences among Proper Orthogonal Decomposition (POD), Singular value decomposition (SVD) and principal component analysis (PCA)?

Proper Orthogonal Decomposition (POD), singular value decomposition (SVD), and principal component analysis (PCA) are three eigenvalue methods used to reduce a high-dimensional data set into fewer ...
Bhanuday Sharma's user avatar

1
2 3 4 5 6