Questions tagged [dimensionality-reduction]

Dimensionality reduction refers to techniques for reducing many variables into a smaller number while keeping as much information as possible. One prominent method is [tag pca]

Filter by
Sorted by
Tagged with
1
vote
1answer
27 views

Do I use the mean vector from my training set to center my testing set when dimension reducing for classification?

Please let me know if this is the right place to ask this (or if any of my tags are wrong) or if I need to write this any differently. Do I use the mean vector from my training set to center my ...
1
vote
0answers
10 views

How effective is Moore Penrose for solving regression problems with overdetermined system of equations?

For regression problems with #Predictors > #observations, I recently read about Moore Penrose (pseudo inverse method) which solves the problem of non invertible matrix in OLS for regression problems. ...
1
vote
0answers
21 views

svd to reduce text matricies

I am having problems with very big texts. after preprocessing each of my docs represented as a matrix of sentences, where each sentence represented as encoded words (each word have a unique vocab ...
1
vote
0answers
43 views

Autoencoder for Dimensionality Reduction - varying result - parameter tuning

I'm not an expert in autoencoders or neural networks by any means, so forgive me if this is a silly question. The problem and steps taken to solve problem are as follows: There exists a data set with ...
0
votes
1answer
21 views

Dimensionality reduction without select components

I would like to use dimensionality reduction algorithm in my pipeline. I have 2k features and I'm using xgboost. My model is rebuilding each day (there are new records that should be involve to ...
3
votes
0answers
20 views

Possible flaw in the MDS method for dimensionality reduction

The MDS (multidimensional scaling) method is used to solve the problem of dimensionality reduction. Basically, it does the following: given $n$ points $x_1,\cdots,x_n\in\mathbb R^d$, try to find a ...
0
votes
1answer
50 views

Low memory error while performing degree 2 polynomial regression on (3000*1835) sized array

I am working on a problem to predict the revenue, a film will generate. Some of the features available in the data set are json collection for the crew, cast which worked in the film. I applied ...
0
votes
1answer
19 views

Potential speedup by applying PCA once on dataset with m rows vs. IncrementalPCA to x batches of size m/x?

I've been working on trying to perform dimensionality reduction on high-dimensional, high-volume datasets (with many rows and columns - around 100,000 - 1M rows and ~500 columns). While the size of ...
0
votes
0answers
22 views

Analogy between Autoencoder and PCA

I know that Autoencoders can be regarded as non-linear generalisations of PCA, but I struggle to understand in depth the analogy between the two. Once PCA has been performed on a function $F(\vec{\...
1
vote
0answers
16 views

Are vectors generated by doc2vec and similar models uniformly distributed?

I have read that vectors in a word2vec model are very much not uniformly distributed and are thought to follow Zipf's law; is this the same for the associated models like paragraph2vec, doc2vec, etc? ...
3
votes
2answers
32 views

find most dense neighborhood of points in high dimensional space

I'm working on a project where I have many high-dimensional points and I want to find the most dense neighborhood of them. Ideally, out of my ~500 points that are each a 4x300 matrix (300 ms time ...
2
votes
1answer
23 views

Magnifying or reducing the size of input groups into a neural network

Say you've got two inputs (X1 and X2) that you want to use to predict Y. You're not sure how important X1 and X2 are for predicting Y, but you assume about even. One-hot encoding is a good strategy ...
2
votes
1answer
82 views

Linear Discriminant Analysis (LDA) before or after k-fold cross-validation?

I have features extracted from a small dataset, would like to reduce the dimensions by using LDA. Also want to do a SVM classification with k-fold cross-validation. My question is: What would be the ...
0
votes
1answer
45 views

comparison of t-SNE and PCA and truncate SVD

How to compare the trucate SVD ,PCA, and T-SNE? What we can say about features if t-SNE and PCA and truncate SVD digaram is in this figure?
0
votes
2answers
18 views

If in t-SNE digaram of binary classification both classes follow the similar curve what does t-SNE diagram show?

If in t-SNE digaram of binary classification both classes follow the similar curve what does t-SNE diagram show for instance: Figure1 or Figure2
2
votes
3answers
96 views

What is the meaning of 2D vectors?

I keep hearing people say something like: lets say you have a 1 dimensional vector of a person that just has his age. Then you add another dimension which is his height, so you have a 2 ...
3
votes
0answers
15 views

High dimensional data stream summarization and processing

Can anyone recommend a method for summarizing and processing high dimensional data streams efficiently and effectively for anomaly detection? In fact, I investigated the different methods for data ...
1
vote
1answer
74 views

Steps on how to use autoencoders to reduce dimensions

I have a dataset that contains text columns. I have used tf-idf to convert those text columns to numerical columns. I want to reduce the dimension of the dataset since tf-idf creates a multitude of ...
0
votes
1answer
56 views

PCA, why variance of eigen values is measure of its utility?

Source - Murphy, 12.3 Heuristic for assessing applicability of PCA. Let the empirical covariance matrix Σ have eigenvalues λ1≥λ2≥···≥λd>0, with mean λ. Explain why the variance of the eigen values, ...
0
votes
1answer
28 views

How to select 500 most pertinents tags among 10000?

Say we have 100,000 documents tagged with 10,000 different tags (Max 5 tag per document). We wish to limit allowed tags to a list of 500 tags. How to select 500 tags in order to cover the largest ...
0
votes
0answers
15 views

Sammon Mapping vs Curvilinear Component Analysis

I am currently busy with a Dimensionality reduction course are are studying a few multi-dimensionality reduction techniques. So both CCA and Sammon Mapping both try give importance to small ...
0
votes
2answers
45 views

How many dimensions should matrix be reduced to with PCA?

I want to reduce the dimensions of my dataset to decrease the complexity without losing too much information. But I am not sure how many dimensions I should reduce to, so that I don't lose too much ...
2
votes
2answers
39 views

How to leverage description data in multi-class classification (dimensionality reduction)

I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as ...
0
votes
0answers
93 views

Regression on list of 2D time-based data

I have data such that: $D=\{(X_1, Y_1, z_1), (X_2, Y_2, z_2), ...,(X_n, Y_n, z_n)\}, n=1000$ Where: $|(X, Y)_k|$ varies in the $[10, 10000]$ range; $x$ are time values; $y$ are values in $T=\{1, 2, ...
0
votes
1answer
82 views

Keras - Autoencoder different from Encoder + Decoder

I build a CNN 1d Autoencoder in Keras, following the advice in this SO question, where Encoder and Decoder are separated. My goal is to re-use the decoder, once the Autoencoder has been trained. The ...
0
votes
0answers
14 views

Can PCA reduce dimensionality of a subset of data whilst still considering the whole set?

I have a nominal set of known data (some 400x400 matrix) and a much larger additional set (~400x40000). Adding each additional column of data-values from the larger set will increase the practical ...
0
votes
0answers
16 views

Suggestions on non-linear dimensionality reduction for small, one-hot encoded dataset

I wish to apply non-linear dimensionality reduction on a very small dataset (less than 100 observations). The dataset is very sparse, of approx 20 columns, each containing either 0 or 1. It's the ...
0
votes
1answer
155 views

'PCA' object has no attribute 'explained_variance_'

Elbow Method - Finding the number of components required to preserve maximum variance. My code: ...
3
votes
2answers
124 views

Why don't we use space filling curves for high-dimensional nearest neighbor search?

Some space filling curves like the Hilbert Curve are able to map an n-dimensional space to a one dimensional line whilst preserving locality. Does that mean that we could map a dataset of high ...
2
votes
1answer
154 views

Document embedding vs locality sensitive hashing for document clustering

I would like to compare two methods: locality sensitivity hashing and document embedding to get the similarity between two documents. Both of those methods encode information of a document in a ...
1
vote
1answer
182 views

Autoencoder or layer-based dimensionality reduction?

I have a few TB of wide data. I want to reduce the number of features in my dataset before feeding my dataset into a classification model... or should I not? Obviously, I will want to try both ...
0
votes
1answer
83 views

Intuition behind the PCA algorithm

I am trying to understand PCA intuitively. Here it goes: After finding the eigenvectors and eigenvalues of the covariance matrix of the dataset, the eigenvalues will represent how spread out the ...
1
vote
0answers
16 views

Clustering of very high dimensional data and large number of examples without losing info in dimensions

I'm trying to get a grasp on scalability of clustering algorithms, and have a toy example in mind. Let's say I have around a million or so songs from $50$ genres. Each song has characteristics - some ...
1
vote
1answer
95 views

How to reduce dimensionality of 3.2B categorical features?

Background: This means a dataset of 7,000 samples and 3.2B columns, which I would have to read into distributed Spark memory somehow. Obviously I want to reduce the number of columns that gets fed ...
0
votes
1answer
52 views

How to structure my data into features and targets for PCA on Big Data?

I want to apply the PCA algorithm from Scikit-Learn.(https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html ) At the part where I have to separate the features and the ...
1
vote
1answer
33 views

Unsupervised Clustering high dimentional data not having estimation for K

I have a dataset (all numerical) of 50K records containing 500 features. we are trying to find fingerprints. Meaning that we would like to cluster the data and report one of the nodes in each cluster ...
1
vote
2answers
44 views

Interpretation of PCA visualisation

I am trying to build a classifier to predict the ratings of a show during a specific time. I have extracted around 109 features, some relating to the time field namely, Day of Year Month of year ...
0
votes
0answers
25 views

What is the dimension reduction method to large numbers of independent features while only two of them are important?why?

What is the dimension reduction method to model a data with large numbers of independent features (for instance 5k features), while only two of them are important (are effective in cost function)? I ...
0
votes
0answers
19 views

What information is encoded in embedding vector lengths?

I have started to investigate word2vec and related embedding strategies. The word2vec training loss is a function of cosine distance and not Euclidean distance. In fact I have been reading various ...
0
votes
0answers
13 views

Dimensionality Reduction. How to explain dynamics of feature subset based on all features data?

I have features: f1..f1000. I want to explain dynamics of particular features subset: f1-f5 based on all features data (based on ...
2
votes
2answers
106 views

Using random forest for selecting variables returns the entire dataframe

I am in the process of dimensionality reduction. I am using Random Forest to find the columns with the highest level of correlation with the target SalePrice column. The problem is that the output ...
0
votes
1answer
24 views

How to use reduced dimensions (of a PCA) for detection purposes?

A general question aiming at the application of a PCA: I want to detect abnormal data points and therefore I want to use a PCA for it at first. The next step is to try several distance functions or ...
0
votes
0answers
14 views

Heuristics, methods to speed up searches over subsets of big set (combinatorially NP hard probably)

I have a reasonable-sized set of size N (say 10 000 objects) in which I am searching for groups of compatible elements. Meaning that I have a function ...
1
vote
1answer
124 views

Which algorithm can be used to reduce dimension of multiple time series?

In my dataset, a data point is essentially a Time series of 6 feature over a year per month so in all, it results in 6*12=72 features. I need to find class outliers so I perform dimensionality ...
0
votes
0answers
70 views

How to perform a 1-way ANOVA right after One-Hot-Encoding

I am at the phase of dimensionality reduction. I am trying to figure out which categorical columns I should keep for my model and which I should discard. Because some of my categorical columns have ...
3
votes
1answer
679 views

How to automate ANOVA in Python

I am at the dimensionality reduction phase of my model. I have a list of categorical columns and I want to find the correlation between each column and my continuous ...
1
vote
1answer
39 views

Eigen Decomposition of Data Matrix for PCA

In PCA we Eigen decompose the covariance matrix, not data matrix, Is it because most data matrices are non-square. If they were, isn't is correct to eigen decompose data matrix than the covariance ...
2
votes
1answer
69 views

When I should use PCA?

I have a data set with 60000 rows and 32 columns. I want to use SVM (with some more constraints that make it more complicated)and I think 32 columns are too large. So I decided to use PCA. But when I ...
1
vote
1answer
43 views

Dimensionality reduction based on value of a variable

I have a dataset including 100k high dimensional data (e.g. houses in LA) (dim=100, e.g. house parameters like area, distance to downtown, etc.). Below is the 2-component PCA representation of the ...
0
votes
1answer
95 views

Does it make sense to train an Autoencoder for Dimensionality Reduction using Mini-Batch Gradient Descent?

I want to reduce the dimensionality of a dataset using a stacked Autoencoder. The size of the dataset and the computing power at my disposal make it very difficult to train the Network using simple, ...