Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [dimensionality-reduction]

Dimensionality reduction refers to techniques for reducing many variables into a smaller number while keeping as much information as possible. One prominent method is [tag pca]

Filter by
Sorted by
Tagged with
1 vote
1 answer
30 views

What is the Purpose of Feature Selection

I have a small medical dataset (200 samples) that contains only 6 cases of the condition I am trying to predict using machine learning. So far, the dataset is not proving useful for predicting the ...
user avatar
  • 333
0 votes
0 answers
13 views

How to Approach Linear Machine-Learning Model When Input Variables are Inconsistent

Disclaimer: I'm relatively new to the data science and ML world -- still trying to get a firm grasp on the fundamentals. I'm trying to overcome a regression challenge involving a large, multi-...
user avatar
0 votes
0 answers
18 views

An autoencoder setup for anomaly detection

I am doing anomaly detection using machine learning. i have tried different models like isolation forest, SVM and KNN. The maximum accuracy that I can get from each of them is $80\%$ accordind to my ...
user avatar
  • 1
-1 votes
2 answers
132 views

Why Do a Set of 3 Clusters Across 1 Dimension and a Set of 3 Clusters Across 2 Dimensions Form 9 Apparent Clusters in 3 Dimensions?

I am sorry if this is a well-known phenomenon but I can't quite wrap my head around this. I have a related question: How To Develop Cluster Models Where the Clusters Occur Along Subsets of Dimensions ...
user avatar
0 votes
0 answers
29 views
+50

Accuracy drops when adding a fully connected layer for dimensionality reduction to a ResNet50

I'm training a ResNet50 for image classification and I'm interested in decreasing the dimensionality of the embedded layer, in order to apply some clustering techniques. The suggested dimension is ...
user avatar
  • 51
0 votes
0 answers
20 views

Would you ever chose t-SNE over UMAP?

UMAP is both faster and captures the global structure better than t-SNE when visualizing high-dimensional data. Is there ever a situation where you would pick t-SNE over UMAP?
user avatar
0 votes
0 answers
14 views

Single scalar from vector

I am aware that this question is very general, but I found this question and it made me curious. What are the sensible ways that you can think of to derive a single scalar value from a vector? Of ...
user avatar
  • 21
0 votes
1 answer
18 views

Dimensionality Reduction of Curved Structural Data

I have been using PCA dimensionality reduction on datasets that are quite linear and now I am tasked with the same on datasets that are largely curved in space. Imagine a noisy sine wave for ...
user avatar
  • 111
0 votes
0 answers
15 views

Clustering Algorithm + Euclidean Distance to find similarities

Goal: Create a tool that recommends similar players based on their statistical profile Process: (1) Standardize data (2) UMAP to reduce dimensionality (c. 50 features) (3) First-Stage Clustering: GMM ...
user avatar
1 vote
1 answer
14 views

How to choose Recursive Feature Elimination parameters

in my project I have >900 features and I thought to use Recursive Feature Elimination algorithm to reduce the dimensionality of my problem (in order to improve the accuracy). But I can't figure out ...
user avatar
0 votes
0 answers
17 views

Searching machine learning algorithm for regression problem with many features

I have a machine learning problem with about 160 features and 400 cases and I want to find the best predictors for a continuous outcome. The dataset contains variables of psychotherapists and clients. ...
user avatar
2 votes
1 answer
40 views

What is the meaning of preserving local or global structure of the data?

I read about PaCMAP dimensionality reduction method (PaCMAP). They wrote that this method preserving both local and global structure of the data in original space. ...
user avatar
  • 267
1 vote
2 answers
45 views

Using PCA for Dimensionality Expansion

I was trying to use t-SNE algorithm for dimensionality reduction and I know this was not the primary usage of this algorithm and not recommended. I saw an implementation here. I am not convinced about ...
user avatar
  • 13
0 votes
0 answers
34 views

PCA in supervised machine learning

I would like to use PCA in supervised ML aiming to generate a binary classification model. The data set I have consists of well validated target variable (labels) concerning the one classification ...
user avatar
1 vote
0 answers
46 views

Can one use PCA to reduce the dimensionality of One-Hot-Encoded data?

I read a couple times that PCA was used as a method to reduce dimensionality for one-hot-encoded data. However, there were also some comments that using PCA is not a good idea since one-hot-encoded ...
user avatar
0 votes
0 answers
9 views

Feature Selection for vector of targets, Y=(y1,...,yn)

If I have an output/target, y, that is a vector of targets, e.g. (y1(t), y2(t), y3(t), y4(t)) and training data X, also a vector, (x1(t),...,xn(t)), and I wish to do e.g. regression, neural nets, and ...
user avatar
  • 21
1 vote
0 answers
11 views

Nearest neighbor face recognition in eigenspace when using dot product of test set with eigenvectors does not match the performance when using sklearn

I am trying to perform Face recognition using PCA (eigenfaces). I have a set of N training images (of dimensions M=wxh), which I have pre-processed into a vertical ...
user avatar
0 votes
0 answers
13 views

Practical significance of number of singular values in SVD

I am working on a binary classification problem. SVD is used for dimensionality reduction and the vector with reduced dimension is used as the feature vector. DNN is used as the classifier. There are ...
user avatar
1 vote
0 answers
19 views

Understanding Linear Discriminant Analysis

I am trying to understand the results of a Linear Discriminant Analysis. For that purpose I used the Iris dataset. I plotted the two first features: Then, I projected these two features in the new ...
user avatar
0 votes
0 answers
15 views

When should I use dimensionality reduction in cluster analysis

I would like to ask a question about dimensionality reduction in cluster analysis. I currently have a table data with 15 rows and 120 columns, and I'll doing cluster analysis on this data. In this ...
user avatar
  • 1
0 votes
0 answers
11 views

How to reduce columns with binary data

I have one-hot encoded some categorical columns and as you might have guessed the resulting data frame is very large and I'm looking for a way to reduce the dimensions of this data frame. PCA is not ...
user avatar
1 vote
2 answers
50 views

Dimensionality reduction for millions of features

I have a dataset with 10 million observations and 1 million sparse features. I would like to build a binary classifier for predicting a particular feature of interest. My main problem is how to deal ...
user avatar
  • 11
0 votes
0 answers
16 views

Isomap/LLE python library that allows precomputed distance matrix

I want apply some algorithm to the distance matrix before reducing dimensionality using Isomap, LLE and Laplacian Eigenmaps. Are there Python libraries I can use for this?
user avatar
2 votes
1 answer
276 views

Can t-SNE be applied to visualize time series datasets

I have multiple time-series datasets containing 9 IMU sensor features. Suppose I use the sliding window method to split all these data into samples with the sequence length of 100, i.e. the dimension ...
user avatar
0 votes
0 answers
18 views

Projection of PCA into new variable space (study area)

I have a set of >30 variables distributed in x,y coordinates across two study areas. I'd like to reduce that dimensionality with a PCA applied in the first study area. Then I'd like to visualize ...
user avatar
0 votes
0 answers
17 views

Best way to classify a high dimension dataset

I am writing a code to classify 4-body systems as 'stable' and 'unstable'. For this purpose, I generated a vast (~1 million) dataset of 4-body systems, and narrowed down the number of possible ...
user avatar
  • 53
0 votes
0 answers
61 views

Variational autoencoder for time series denoising and dimentionality reduction

I have a dataset X of multiple series say 100 (size=100). I would like to use VAE to both denoise the data and reduce the dimensions to a smaller latent space Z (size Z << size X), because I ...
user avatar
0 votes
0 answers
7 views

How to interpret data projected on the sum of first few principal components weighted by eigen values?

I have simulation time series data of a molecule from Molecular dynamics and I want to visualize the very high-dimensional trajectory in two dimensions and also identify some clusters. The problem is ...
user avatar
  • 1
0 votes
0 answers
38 views

PCA vs t-SNE in asset pricing

So I am trying dimensionality reduction techniques on the S&P500 FY2020 data. I understand the CAPM model and the fact that doing a PCA determines my market variability factor (the first PCA ...
user avatar
1 vote
1 answer
30 views

Factor Analysis vs PCA

Could someone please explain when FA is used or when PCA is used, as I understood FA do dimensionality reduction, however PCA - the main goal is the same. Then which one should I use and in which ...
user avatar
  • 33
0 votes
0 answers
14 views

How to classify (supervised) a multi dimensional vector?

What kinds of machine learning tool is used to classify a vector of data which are not spatially correlated? I have a 158*158 image*15000 samples which I tried to ...
user avatar
1 vote
1 answer
44 views

Reusing UMAP transformation

I have this use case: I want to apply a dimension reduction with UMAP to a initial dataset of high-dimension vectors (100d), and later, in second place, have the oppurtinity to add new data points ...
user avatar
1 vote
0 answers
274 views

What are the differences among Proper Orthogonal Decomposition (POD), Singular value decomposition (SVD) and principal component analysis (PCA)?

Proper Orthogonal Decomposition (POD), singular value decomposition (SVD), and principal component analysis (PCA) are three eigenvalue methods used to reduce a high-dimensional data set into fewer ...
user avatar
0 votes
0 answers
37 views

How to implement the contribution analysis using PCA?

I have been looking into implementing the Q-Residual and Hotelling's T statistics calculation to the PCA components which is similar to the following article and website: Structural Health Monitoring ...
user avatar
0 votes
1 answer
36 views

Latent Semantic Analysis on image features using k means

I have extracted Color Moments from a set of images and want to use Kmeans to perform dimensionality reduction and find the top k latent semantics. How can I use Kmeans for latent semantic analysis?
user avatar
0 votes
0 answers
30 views

How to decide number of hidden layers and number of neurons for Autoencoder for dimensionality reduction function?

I have been looking into deep learning and what caught my attention is the implementation of Autoencoder as a dimensionality reduction function for anomaly detection. I found out about it through the ...
user avatar
4 votes
1 answer
196 views

What are the main differences between uwot and umap packages in R?

There are two packages in R that implement the UMAP algorithm for low-dimensional embedding ('uwot' and 'umap'). I've found they can give vastly different results for some datasets. For example, the ...
user avatar
1 vote
1 answer
79 views

Encoding very large dataset to one-hot encoding matrix

I have a dataset of text corpus where the unique characters in the text are around 400. The maximum row length is 3000. We have 20000 rows, so we would have like $2000\times3000\times400$ one-hot ...
user avatar
  • 179
0 votes
1 answer
31 views

Is t-SNE good at clustering instances with the same trend?

I have a dataset of time-series data with 50k examples and a length of 90, like the images showed below: I was wondering whether t-SNE or any type of dimensionality reduction could group the ...
user avatar
0 votes
0 answers
53 views

LDA transform does produce same result on testing data as fit_transform on training data

I have a data set with over 2000 variables where I am using LDA (Linear discriminant analysis) for dimension reduction with the intention of having maximum class separability. However, LDA fails to ...
user avatar
1 vote
0 answers
24 views

TSNE parameters

Trying to tune the parameters of sklearn.manifold.TSNE(n_components=2, *, perplexity=30.0, early_exaggeration=12.0, learning_rate=200.0, n_iter=1000, n_iter_without_progress=300, min_grad_norm=1e-07, ...
user avatar
  • 37
1 vote
0 answers
21 views

Comparing the similarity structure of 2 distance matrices (computed from sentence embedding)

I apologize if this question lacks clarity, my mathematical background on the topic is limited and was hoping to find some guidance. I would like to compare 2 distance matrices that contain pair-wise ...
user avatar
  • 11
0 votes
0 answers
39 views

Numpy-like reduction of particular dimensions in Pandas DataFrame?

With a numpy multidimensional arrays, it is easy to reduce particular axes with functions such as np.mean or np.sum. For example,...
user avatar
0 votes
0 answers
11 views

Any reliable dimension reduction implementations available to address class overlapping scenario?

I am currently resolving a class overlapping problem in machine learning and while running some class separation experiments I have observed that Linear Discriminant Analysis (LDA) is able to perform ...
user avatar
0 votes
0 answers
10 views

Handling High dimensionality datasets for EDA

I have a dataset consists of 500k rows and around 500 variables initially. And I would like to run EDA among these, then modeling of course. But this high dimension should be reduced of course. First ...
user avatar
1 vote
2 answers
33 views

Dimensionality reduction of vectors with null values

I have vectors of same length where each entry can have the value 0, 1 or null. V = {[0,1,1,1,null,0], [null,1,0,null,0,1], ...} How can I perform a dimensionality ...
user avatar
  • 111
0 votes
0 answers
9 views

Embedding data with a graphical structure

I have an $n\times p$ dataset and wish to embed each observation in a $d$ dimensional space. The trouble is, my predictors are derived from a DAG. For a simplified example, suppose the DAG is as ...
user avatar
0 votes
0 answers
34 views

feature selection for categorical variables

I have been working on this issue for quite a while and going nowhere. If I have categorical features in my dataset and some of them have high dimensions, if I OHE them, I get a dataset with high ...
user avatar
  • 1,241
2 votes
0 answers
58 views

Theoretical differences between KPCA and t-SNE?

I (think I) understand the underlying principles of most dimensionality reduction methods (MDS, IsoMap, t-SNE, Spectral Embedding, Diffusion maps, etc...). Some of the algorithms I use the most are ...
user avatar
  • 21
0 votes
0 answers
651 views

Why am I getting a different answer in Principal Component Analysis dimensional reduction?

Problem-: Consider the two dimensional patterns (2, 1), (3, 5), (4, 3), (5, 6), (6, 7), (7, 8). Compute the principal component using PCA Algorithm. Use PCA Algorithm to transform the pattern (2, 1) ...
user avatar
  • 31

1
2 3 4 5 6