The 2024 Developer Survey results are live! See the results

# Questions tagged [dimensionality-reduction]

Dimensionality reduction refers to techniques for reducing many variables into a smaller number while keeping as much information as possible. One prominent method is [tag pca]

291 questions
Filter by
Sorted by
Tagged with
13 views

### How can discrete wavelet transform (DWT) help in data reduction

Hi I am reading Data Mining concepts and Techniques by jiawei Han. In page 100 it is mentioned that Wavelet transform helps in data reduction. But I can not find the mathematical proof of this as well ...
15 views

• 13
45 views

### Determining "filters" dimension after a convolution operation

I tried to calculate the "filtered" dimension and I seem to be getting it wrong. Below there is the image I am trying to calculate the "filtered" dimension for, where you have 192 ...
1 vote
148 views

### Clustering by using Locality sensitive hashing *after* Random projection

It is well known that Random Projection (RP) is tightly linked to Locality Sensitive Hashing (LSH). My goal is to cluster a large number of points lying in a d-dimensional Euclidean space, where $d$ ...
92 views

### Is it possible to use text Auto Encoders without text generation?

I have a use case where I have large texts, and a lot of it. Pretty often the sequence length exceeds 1000 tokens. I need a lower dimensional compression of the texts as an input for a classifier. The ...
• 101
16 views

### Reducing the dimensionality of entire dataset

I have a lot of datasets with different shapes. For example few of them are (90, 892), (74, 853), (93, 765), ... etc. I want to convert this shape to (90, 4), (74, 4), (93, 4) ... (x, 4). And, after ...
1 vote
133 views

### How to describe this UMAP connectivity figure

I generated this UMAP connectivity diagram of my research data. How do I interpret/describe this plot regarding the UMAP connectivity? Is it correct to say that: As there is a lot of connectivity ...
• 147
59 views

### Classification problem with no context in numerical features

I have an extremely abstract and numeric data with equally abstract objective. I have around 3000 rows of train data (df_train), where I have a binary target ...
• 118
83 views

### How to understand to what maximum size you can reduce the dimension of data and avoid the curse of the dimensionality?

i have a question, maybe someone could help me. I use t-sne (also tried umap) to reduce the dimensionality of the text embeddings dataset (size of embedding 300). after that I will cluster using ...
• 183
27 views

### How can "profiling" be done in a dataset? What are the different techniques?

I am currently doing an analysis in which I need to "profile" each record. For example, let's say I have a dataset of accounts with customer information (name, id, address, money spent, ...
20 views

### PCA, Better performances with 300 components rather than 400 components : why?

I am building this content based image retrieval system. I basically extract feature maps of size 1024x1x1 using any backbone. I then proceed to apply PCA on the extracted features in order to ...
1 vote
176 views

### What is the Purpose of Feature Selection

I have a small medical dataset (200 samples) that contains only 6 cases of the condition I am trying to predict using machine learning. So far, the dataset is not proving useful for predicting the ...
• 437
41 views

### How to Approach Linear Machine-Learning Model When Input Variables are Inconsistent

Disclaimer: I'm relatively new to the data science and ML world -- still trying to get a firm grasp on the fundamentals. I'm trying to overcome a regression challenge involving a large, multi-...
138 views

### An autoencoder setup for anomaly detection

I am doing anomaly detection using machine learning. i have tried different models like isolation forest, SVM and KNN. The maximum accuracy that I can get from each of them is $80\%$ accordind to my ...
• 11
165 views

### Why Do a Set of 3 Clusters Across 1 Dimension and a Set of 3 Clusters Across 2 Dimensions Form 9 Apparent Clusters in 3 Dimensions?

I am sorry if this is a well-known phenomenon but I can't quite wrap my head around this. I have a related question: How To Develop Cluster Models Where the Clusters Occur Along Subsets of Dimensions ...
1 vote
37 views

### Accuracy drops when adding a fully connected layer for dimensionality reduction to a ResNet50

I'm training a ResNet50 for image classification and I'm interested in decreasing the dimensionality of the embedded layer, in order to apply some clustering techniques. The suggested dimension is ...
• 61
22 views

### Single scalar from vector

I am aware that this question is very general, but I found this question and it made me curious. What are the sensible ways that you can think of to derive a single scalar value from a vector? Of ...
• 21
29 views

### Dimensionality Reduction of Curved Structural Data

I have been using PCA dimensionality reduction on datasets that are quite linear and now I am tasked with the same on datasets that are largely curved in space. Imagine a noisy sine wave for ...
• 111
1 vote
27 views

### How to choose Recursive Feature Elimination parameters

in my project I have >900 features and I thought to use Recursive Feature Elimination algorithm to reduce the dimensionality of my problem (in order to improve the accuracy). But I can't figure out ...
34 views

### Searching machine learning algorithm for regression problem with many features

I have a machine learning problem with about 160 features and 400 cases and I want to find the best predictors for a continuous outcome. The dataset contains variables of psychotherapists and clients. ...
2k views

### What is the meaning of preserving local or global structure of the data?

I read about PaCMAP dimensionality reduction method (PaCMAP). They wrote that this method preserving both local and global structure of the data in original space. ...
• 307
1 vote
233 views

### Using PCA for Dimensionality Expansion

I was trying to use t-SNE algorithm for dimensionality reduction and I know this was not the primary usage of this algorithm and not recommended. I saw an implementation here. I am not convinced about ...
• 13
998 views

### Can one use PCA to reduce the dimensionality of One-Hot-Encoded data?

I read a couple times that PCA was used as a method to reduce dimensionality for one-hot-encoded data. However, there were also some comments that using PCA is not a good idea since one-hot-encoded ...
1 vote
19 views

### Nearest neighbor face recognition in eigenspace when using dot product of test set with eigenvectors does not match the performance when using sklearn

I am trying to perform Face recognition using PCA (eigenfaces). I have a set of N training images (of dimensions M=wxh), which I have pre-processed into a vertical ...
• 111
1 vote
37 views

### Understanding Linear Discriminant Analysis

I am trying to understand the results of a Linear Discriminant Analysis. For that purpose I used the Iris dataset. I plotted the two first features: Then, I projected these two features in the new ...
1 vote
361 views

### Dimensionality reduction for millions of features

I have a dataset with 10 million observations and 1 million sparse features. I would like to build a binary classifier for predicting a particular feature of interest. My main problem is how to deal ...
• 11
3k views

### Can t-SNE be applied to visualize time series datasets

I have multiple time-series datasets containing 9 IMU sensor features. Suppose I use the sliding window method to split all these data into samples with the sequence length of 100, i.e. the dimension ...
24 views

### Projection of PCA into new variable space (study area)

I have a set of >30 variables distributed in x,y coordinates across two study areas. I'd like to reduce that dimensionality with a PCA applied in the first study area. Then I'd like to visualize ...
162 views

### PCA vs t-SNE in asset pricing

So I am trying dimensionality reduction techniques on the S&P500 FY2020 data. I understand the CAPM model and the fact that doing a PCA determines my market variability factor (the first PCA ...
1 vote
609 views

### Factor Analysis vs PCA

Could someone please explain when FA is used or when PCA is used, as I understood FA do dimensionality reduction, however PCA - the main goal is the same. Then which one should I use and in which ...
• 33
1 vote