Questions tagged [dimensionality-reduction]

Dimensionality reduction refers to techniques for reducing many variables into a smaller number while keeping as much information as possible. One prominent method is [tag pca]

Filter by
Sorted by
Tagged with
0
votes
0answers
17 views

Dimensionality Reduction on Cross Channel EEG data

I'm working on a project predicting seizures events using 10 minute EEG data recordings. This involves identifying the preictal (pre-seizure) phase from the interictal (normal/between seizure) phase, ...
0
votes
0answers
6 views

Factor Analysis with Mixed Data gives many components

I performed Factor Analysis with Mixed Data using PCAmixdata package from R. My dataset consists of 115000 records with 40 features of both categorical and continuous data. I checked the eigenvalues ...
1
vote
0answers
12 views

Factor Analysis with Mixed Data Concurrent Approach with PCAmixdata in R

I am trying to perform Factor Analysis over Mixed Data using R with PCAmixdata package. My dataset is huge with almost 115000 records and almost 40 features of both categorical and continuous. When I ...
1
vote
1answer
26 views

How do i perform feature selection for clustering?

I am new to Data science and trying to learn clustering? I have to partition the given dataset into different clusters into customer clusters based on their purchasing habits? How do I select the ...
0
votes
2answers
29 views

Need suggestions on customer segmentation

I have been tasked with performing customer segmentation for a Business to business use case based on customer purchase history. Can experts provide me inputs on how do I proceed with customer ...
0
votes
1answer
19 views

The OLAP (On-Line Analytical Processing) cube with 4 dimensions [closed]

A typical OLAP cube looks like this: As I can see, this cube can work with 2 or 3 dimensions, but what if I have 4 dimensions to produce facts? Should I use star schema instead when having more than ...
2
votes
2answers
57 views

PCA for complex-valued data

I'm quite shocked for encountering this error on PCA from sklearn ValueError: Complex data not supported After trying to fit ...
0
votes
0answers
10 views

Using the input record's distribution as a composite input feature

I have a very interesting dataset that I need to use for doing regression. It is production data from stainless steel production and I have about 290 input features, so I need to start reducing the ...
1
vote
2answers
22 views

PCA vs.KernelPCA: which one to use for high dimensional data?

I have a dataset which contains a lot of features (>>3). For computational reasons, I would like to apply a dimensionality reduction. At this point I could use different techniques: standard PCA ...
0
votes
1answer
21 views

Should dimensionality reduction be done before k-means clustering if there are many features?

My data contains over 200 features and over 500 observations. I want to place the observations into a number of clusters based on the features that make them different. There are numerous ideas I ...
0
votes
0answers
4 views

Feature Selection Using Linear Discriminant Analysis

In this post it says that when input variables are continuous and response is categorical, in that case we can use Linear Discriminant Analysis (LDA). But as far as i know it is a dimentionality ...
0
votes
0answers
2 views

PCA targeted away from some subspace

Is there an existing technique allowing to do PCA maximizing not the variance per se, but the variance away from some direction? Imagine I have high-dim data with two different labels L1, L2 and I ...
0
votes
0answers
9 views

Supervised dimensionality reduction for multilabel data

re there algorithms for supervised dimensionality reduction like Linear Discriminant Analysis (LDA) for multilabel classification? If I understood it right, the implementation of LDA in scikit-learn ...
0
votes
1answer
51 views

How to perform feature selection with Categorical Variables and Continuous Target, provided that data is not normally distributed?

Basically I am trying a use Multi Linear Regression Model to predict the salaries of employees. I have a total of 88 dependent feature from which 19 are categorical and the rest are continuous. I have ...
0
votes
0answers
11 views

Dimension reduction using non-linear PCA

I am working on an undergraduate astronomy research in which we are analyzing geometrical complexities of different sattelite images of man-made and natural structures on Earth. The different images ...
1
vote
1answer
27 views

Do I use the mean vector from my training set to center my testing set when dimension reducing for classification?

Please let me know if this is the right place to ask this (or if any of my tags are wrong) or if I need to write this any differently. Do I use the mean vector from my training set to center my ...
1
vote
0answers
12 views

How effective is Moore Penrose for solving regression problems with overdetermined system of equations?

For regression problems with #Predictors > #observations, I recently read about Moore Penrose (pseudo inverse method) which solves the problem of non invertible matrix in OLS for regression problems. ...
1
vote
0answers
23 views

svd to reduce text matricies

I am having problems with very big texts. after preprocessing each of my docs represented as a matrix of sentences, where each sentence represented as encoded words (each word have a unique vocab ...
1
vote
0answers
83 views

Autoencoder for Dimensionality Reduction - varying result - parameter tuning

I'm not an expert in autoencoders or neural networks by any means, so forgive me if this is a silly question. The problem and steps taken to solve problem are as follows: There exists a data set with ...
0
votes
1answer
23 views

Dimensionality reduction without select components

I would like to use dimensionality reduction algorithm in my pipeline. I have 2k features and I'm using xgboost. My model is rebuilding each day (there are new records that should be involve to ...
3
votes
0answers
20 views

Possible flaw in the MDS method for dimensionality reduction

The MDS (multidimensional scaling) method is used to solve the problem of dimensionality reduction. Basically, it does the following: given $n$ points $x_1,\cdots,x_n\in\mathbb R^d$, try to find a ...
0
votes
1answer
179 views

Low memory error while performing degree 2 polynomial regression on (3000*1835) sized array

I am working on a problem to predict the revenue, a film will generate. Some of the features available in the data set are json collection for the crew, cast which worked in the film. I applied ...
0
votes
1answer
24 views

Potential speedup by applying PCA once on dataset with m rows vs. IncrementalPCA to x batches of size m/x?

I've been working on trying to perform dimensionality reduction on high-dimensional, high-volume datasets (with many rows and columns - around 100,000 - 1M rows and ~500 columns). While the size of ...
0
votes
0answers
24 views

Analogy between Autoencoder and PCA

I know that Autoencoders can be regarded as non-linear generalisations of PCA, but I struggle to understand in depth the analogy between the two. Once PCA has been performed on a function $F(\vec{\...
1
vote
0answers
17 views

Are vectors generated by doc2vec and similar models uniformly distributed?

I have read that vectors in a word2vec model are very much not uniformly distributed and are thought to follow Zipf's law; is this the same for the associated models like paragraph2vec, doc2vec, etc? ...
3
votes
2answers
37 views

find most dense neighborhood of points in high dimensional space

I'm working on a project where I have many high-dimensional points and I want to find the most dense neighborhood of them. Ideally, out of my ~500 points that are each a 4x300 matrix (300 ms time ...
2
votes
1answer
24 views

Magnifying or reducing the size of input groups into a neural network

Say you've got two inputs (X1 and X2) that you want to use to predict Y. You're not sure how important X1 and X2 are for predicting Y, but you assume about even. One-hot encoding is a good strategy ...
3
votes
3answers
201 views

Linear Discriminant Analysis (LDA) before or after k-fold cross-validation?

I have features extracted from a small dataset, would like to reduce the dimensions by using LDA. Also want to do a SVM classification with k-fold cross-validation. My question is: What would be the ...
1
vote
1answer
102 views

comparison of t-SNE and PCA and truncate SVD

How to compare the trucate SVD ,PCA, and T-SNE? What we can say about features if t-SNE and PCA and truncate SVD digaram is in this figure?
0
votes
2answers
29 views

If in t-SNE digaram of binary classification both classes follow the similar curve what does t-SNE diagram show?

If in t-SNE digaram of binary classification both classes follow the similar curve what does t-SNE diagram show for instance: Figure1 or Figure2
2
votes
3answers
160 views

What is the meaning of 2D vectors?

I keep hearing people say something like: lets say you have a 1 dimensional vector of a person that just has his age. Then you add another dimension which is his height, so you have a 2 ...
3
votes
0answers
19 views

High dimensional data stream summarization and processing

Can anyone recommend a method for summarizing and processing high dimensional data streams efficiently and effectively for anomaly detection? In fact, I investigated the different methods for data ...
1
vote
1answer
91 views

Steps on how to use autoencoders to reduce dimensions

I have a dataset that contains text columns. I have used tf-idf to convert those text columns to numerical columns. I want to reduce the dimension of the dataset since tf-idf creates a multitude of ...
0
votes
1answer
83 views

PCA, why variance of eigen values is measure of its utility?

Source - Murphy, 12.3 Heuristic for assessing applicability of PCA. Let the empirical covariance matrix Σ have eigenvalues λ1≥λ2≥···≥λd>0, with mean λ. Explain why the variance of the eigen values, ...
0
votes
1answer
28 views

How to select 500 most pertinents tags among 10000?

Say we have 100,000 documents tagged with 10,000 different tags (Max 5 tag per document). We wish to limit allowed tags to a list of 500 tags. How to select 500 tags in order to cover the largest ...
0
votes
0answers
16 views

Sammon Mapping vs Curvilinear Component Analysis

I am currently busy with a Dimensionality reduction course are are studying a few multi-dimensionality reduction techniques. So both CCA and Sammon Mapping both try give importance to small ...
0
votes
2answers
82 views

How many dimensions should matrix be reduced to with PCA?

I want to reduce the dimensions of my dataset to decrease the complexity without losing too much information. But I am not sure how many dimensions I should reduce to, so that I don't lose too much ...
2
votes
2answers
47 views

How to leverage description data in multi-class classification (dimensionality reduction)

I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as ...
0
votes
0answers
93 views

Regression on list of 2D time-based data

I have data such that: $D=\{(X_1, Y_1, z_1), (X_2, Y_2, z_2), ...,(X_n, Y_n, z_n)\}, n=1000$ Where: $|(X, Y)_k|$ varies in the $[10, 10000]$ range; $x$ are time values; $y$ are values in $T=\{1, 2, ...
1
vote
1answer
148 views

Keras - Autoencoder different from Encoder + Decoder

I build a CNN 1d Autoencoder in Keras, following the advice in this SO question, where Encoder and Decoder are separated. My goal is to re-use the decoder, once the Autoencoder has been trained. The ...
0
votes
0answers
16 views

Can PCA reduce dimensionality of a subset of data whilst still considering the whole set?

I have a nominal set of known data (some 400x400 matrix) and a much larger additional set (~400x40000). Adding each additional column of data-values from the larger set will increase the practical ...
0
votes
0answers
16 views

Suggestions on non-linear dimensionality reduction for small, one-hot encoded dataset

I wish to apply non-linear dimensionality reduction on a very small dataset (less than 100 observations). The dataset is very sparse, of approx 20 columns, each containing either 0 or 1. It's the ...
0
votes
1answer
440 views

'PCA' object has no attribute 'explained_variance_'

Elbow Method - Finding the number of components required to preserve maximum variance. My code: ...
3
votes
2answers
168 views

Why don't we use space filling curves for high-dimensional nearest neighbor search?

Some space filling curves like the Hilbert Curve are able to map an n-dimensional space to a one dimensional line whilst preserving locality. Does that mean that we could map a dataset of high ...
2
votes
1answer
274 views

Document embedding vs locality sensitive hashing for document clustering

I would like to compare two methods: locality sensitivity hashing and document embedding to get the similarity between two documents. Both of those methods encode information of a document in a ...
1
vote
1answer
566 views

Autoencoder or layer-based dimensionality reduction?

I have a few TB of wide data. I want to reduce the number of features in my dataset before feeding my dataset into a classification model... or should I not? Obviously, I will want to try both ...
0
votes
1answer
156 views

Intuition behind the PCA algorithm

I am trying to understand PCA intuitively. Here it goes: After finding the eigenvectors and eigenvalues of the covariance matrix of the dataset, the eigenvalues will represent how spread out the ...
1
vote
0answers
17 views

Clustering of very high dimensional data and large number of examples without losing info in dimensions

I'm trying to get a grasp on scalability of clustering algorithms, and have a toy example in mind. Let's say I have around a million or so songs from $50$ genres. Each song has characteristics - some ...
1
vote
1answer
123 views

How to reduce dimensionality of 3.2B categorical features?

Background: This means a dataset of 7,000 samples and 3.2B columns, which I would have to read into distributed Spark memory somehow. Obviously I want to reduce the number of columns that gets fed ...
0
votes
1answer
71 views

How to structure my data into features and targets for PCA on Big Data?

I want to apply the PCA algorithm from Scikit-Learn.(https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html ) At the part where I have to separate the features and the ...

1
2 3 4 5