Questions tagged [dimensionality-reduction]

Dimensionality reduction refers to techniques for reducing many variables into a smaller number while keeping as much information as possible. One prominent method is [tag pca]

Filter by
Sorted by
Tagged with
0
votes
1answer
21 views

comparison of t-SNE and PCA and truncate SVD

How to compare the trucate SVD ,PCA, and T-SNE? What we can say about features if t-SNE and PCA and truncate SVD digaram is in this figure?
0
votes
2answers
14 views

If in t-SNE digaram of binary classification both classes follow the similar curve what does t-SNE diagram show?

If in t-SNE digaram of binary classification both classes follow the similar curve what does t-SNE diagram show for instance: Figure1 or Figure2
2
votes
3answers
40 views

What is the meaning of 2D vectors?

I keep hearing people say something like: lets say you have a 1 dimensional vector of a person that just has his age. Then you add another dimension which is his height, so you have a 2 ...
3
votes
0answers
14 views

High dimensional data stream summarization and processing

Can anyone recommend a method for summarizing and processing high dimensional data streams efficiently and effectively for anomaly detection? In fact, I investigated the different methods for data ...
1
vote
1answer
63 views

Steps on how to use autoencoders to reduce dimensions

I have a dataset that contains text columns. I have used tf-idf to convert those text columns to numerical columns. I want to reduce the dimension of the dataset since tf-idf creates a multitude of ...
0
votes
1answer
39 views

PCA, why variance of eigen values is measure of its utility?

Source - Murphy, 12.3 Heuristic for assessing applicability of PCA. Let the empirical covariance matrix Σ have eigenvalues λ1≥λ2≥···≥λd>0, with mean λ. Explain why the variance of the eigen values, ...
0
votes
1answer
25 views

How to select 500 most pertinents tags among 10000?

Say we have 100,000 documents tagged with 10,000 different tags (Max 5 tag per document). We wish to limit allowed tags to a list of 500 tags. How to select 500 tags in order to cover the largest ...
0
votes
0answers
14 views

Sammon Mapping vs Curvilinear Component Analysis

I am currently busy with a Dimensionality reduction course are are studying a few multi-dimensionality reduction techniques. So both CCA and Sammon Mapping both try give importance to small ...
0
votes
2answers
29 views

How many dimensions should matrix be reduced to with PCA?

I want to reduce the dimensions of my dataset to decrease the complexity without losing too much information. But I am not sure how many dimensions I should reduce to, so that I don't lose too much ...
2
votes
2answers
38 views

How to leverage description data in multi-class classification (dimensionality reduction)

I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as ...
0
votes
0answers
93 views

Regression on list of 2D time-based data

I have data such that: $D=\{(X_1, Y_1, z_1), (X_2, Y_2, z_2), ...,(X_n, Y_n, z_n)\}, n=1000$ Where: $|(X, Y)_k|$ varies in the $[10, 10000]$ range; $x$ are time values; $y$ are values in $T=\{1, 2, ...
0
votes
1answer
45 views

Keras - Autoencoder different from Encoder + Decoder

I build a CNN 1d Autoencoder in Keras, following the advice in this SO question, where Encoder and Decoder are separated. My goal is to re-use the decoder, once the Autoencoder has been trained. The ...
0
votes
0answers
12 views

Can PCA reduce dimensionality of a subset of data whilst still considering the whole set?

I have a nominal set of known data (some 400x400 matrix) and a much larger additional set (~400x40000). Adding each additional column of data-values from the larger set will increase the practical ...
0
votes
0answers
15 views

Suggestions on non-linear dimensionality reduction for small, one-hot encoded dataset

I wish to apply non-linear dimensionality reduction on a very small dataset (less than 100 observations). The dataset is very sparse, of approx 20 columns, each containing either 0 or 1. It's the ...
0
votes
1answer
57 views

'PCA' object has no attribute 'explained_variance_'

Elbow Method - Finding the number of components required to preserve maximum variance. My code: ...
3
votes
2answers
87 views

Why don't we use space filling curves for high-dimensional nearest neighbor search?

Some space filling curves like the Hilbert Curve are able to map an n-dimensional space to a one dimensional line whilst preserving locality. Does that mean that we could map a dataset of high ...
2
votes
1answer
66 views

Document embedding vs locality sensitive hashing for document clustering

I would like to compare two methods: locality sensitivity hashing and document embedding to get the similarity between two documents. Both of those methods encode information of a document in a ...
0
votes
1answer
40 views

Autoencoder or layer-based dimensionality reduction?

I have a few TB of wide data. I want to reduce the number of features in my dataset before feeding my dataset into a classification model... or should I not? Obviously, I will want to try both ...
0
votes
1answer
48 views

Intuition behind the PCA algorithm

I am trying to understand PCA intuitively. Here it goes: After finding the eigenvectors and eigenvalues of the covariance matrix of the dataset, the eigenvalues will represent how spread out the ...
1
vote
0answers
14 views

Clustering of very high dimensional data and large number of examples without losing info in dimensions

I'm trying to get a grasp on scalability of clustering algorithms, and have a toy example in mind. Let's say I have around a million or so songs from $50$ genres. Each song has characteristics - some ...
1
vote
1answer
70 views

How to reduce dimensionality of 3.2B categorical features?

Background: This means a dataset of 7,000 samples and 3.2B columns, which I would have to read into distributed Spark memory somehow. Obviously I want to reduce the number of columns that gets fed ...
0
votes
1answer
39 views

How to structure my data into features and targets for PCA on Big Data?

I want to apply the PCA algorithm from Scikit-Learn.(https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html ) At the part where I have to separate the features and the ...
1
vote
1answer
33 views

Unsupervised Clustering high dimentional data not having estimation for K

I have a dataset (all numerical) of 50K records containing 500 features. we are trying to find fingerprints. Meaning that we would like to cluster the data and report one of the nodes in each cluster ...
1
vote
2answers
40 views

Interpretation of PCA visualisation

I am trying to build a classifier to predict the ratings of a show during a specific time. I have extracted around 109 features, some relating to the time field namely, Day of Year Month of year ...
0
votes
0answers
25 views

What is the dimension reduction method to large numbers of independent features while only two of them are important?why?

What is the dimension reduction method to model a data with large numbers of independent features (for instance 5k features), while only two of them are important (are effective in cost function)? I ...
0
votes
0answers
16 views

What information is encoded in embedding vector lengths?

I have started to investigate word2vec and related embedding strategies. The word2vec training loss is a function of cosine distance and not Euclidean distance. In fact I have been reading various ...
0
votes
0answers
12 views

Dimensionality Reduction. How to explain dynamics of feature subset based on all features data?

I have features: f1..f1000. I want to explain dynamics of particular features subset: f1-f5 based on all features data (based on ...
2
votes
2answers
101 views

Using random forest for selecting variables returns the entire dataframe

I am in the process of dimensionality reduction. I am using Random Forest to find the columns with the highest level of correlation with the target SalePrice column. The problem is that the output ...
0
votes
1answer
23 views

How to use reduced dimensions (of a PCA) for detection purposes?

A general question aiming at the application of a PCA: I want to detect abnormal data points and therefore I want to use a PCA for it at first. The next step is to try several distance functions or ...
0
votes
0answers
14 views

Heuristics, methods to speed up searches over subsets of big set (combinatorially NP hard probably)

I have a reasonable-sized set of size N (say 10 000 objects) in which I am searching for groups of compatible elements. Meaning that I have a function ...
1
vote
1answer
88 views

Which algorithm can be used to reduce dimension of multiple time series?

In my dataset, a data point is essentially a Time series of 6 feature over a year per month so in all, it results in 6*12=72 features. I need to find class outliers so I perform dimensionality ...
0
votes
0answers
46 views

How to perform a 1-way ANOVA right after One-Hot-Encoding

I am at the phase of dimensionality reduction. I am trying to figure out which categorical columns I should keep for my model and which I should discard. Because some of my categorical columns have ...
3
votes
1answer
444 views

How to automate ANOVA in Python

I am at the dimensionality reduction phase of my model. I have a list of categorical columns and I want to find the correlation between each column and my continuous ...
1
vote
1answer
39 views

Eigen Decomposition of Data Matrix for PCA

In PCA we Eigen decompose the covariance matrix, not data matrix, Is it because most data matrices are non-square. If they were, isn't is correct to eigen decompose data matrix than the covariance ...
2
votes
1answer
58 views

When I should use PCA?

I have a data set with 60000 rows and 32 columns. I want to use SVM (with some more constraints that make it more complicated)and I think 32 columns are too large. So I decided to use PCA. But when I ...
1
vote
1answer
39 views

Dimensionality reduction based on value of a variable

I have a dataset including 100k high dimensional data (e.g. houses in LA) (dim=100, e.g. house parameters like area, distance to downtown, etc.). Below is the 2-component PCA representation of the ...
0
votes
1answer
57 views

Does it make sense to train an Autoencoder for Dimensionality Reduction using Mini-Batch Gradient Descent?

I want to reduce the dimensionality of a dataset using a stacked Autoencoder. The size of the dataset and the computing power at my disposal make it very difficult to train the Network using simple, ...
0
votes
0answers
19 views

Design / Choice of Autoencoder to classify temporal pattern in images

Suppose I have a temporal stack of images of shape $m \times n \times k$ where shape of each image is $m \times n$ and $k$ represents the temporal dimension. In this context, I am trying to detect and ...
0
votes
1answer
116 views

Implication of a dominant Principal Component in PCA analysis

I need help, are there any practical implications of a dominant principal component. For example, if of three PCs, PC1 explains almost 100% of the variance in this dataset, What does this mean in ...
1
vote
2answers
122 views

Difference between LASSO penalty in neural network and just LASSO regression

I wonder whether those two have any significant differences. I think in neural network, the lasso penalty put on the loss function makes the model simpler and introduces more sparsity by ...
1
vote
1answer
260 views

How to calculate compression ratio when using autoencoder in neural network

For example, if I use an autoencoder to compress a 1000 dimensional data set to 25 dimensions. Is the compression ratio is 40:1? Other info: The dataset contains 5000 samples. 2 million parameters ...
5
votes
1answer
1k views

What is the difference between and Embedding Layer and an Autoencoder?

I'm reading about Embedding layers, especially applied to NLP and word2vec, and they seem nothing more than an application of Autoencoders for dimensionality reduction. Are they different? If so, what ...
1
vote
1answer
72 views

Feature selection or Dimension reduction in unsupervised learning

I'm trying to do Embedded clustering using kmeans. This is customer data, so it involves a lot of sentences, so I'm using the universal sentence encoder before clustering. But I should be doing a ...
2
votes
1answer
39 views

How to scale or standardize data that is mostly 0 (ranges from 0-1)?

I am relatively new to data science and big data munging in general. I currently have various columns of data that range from $0-1$, but most of the values in each ...
2
votes
1answer
69 views

Multiclass classification with high number of classes, high number of features and small sample size

I am working on a biology related dataset with over 300K features, and I only have about 5K samples. I want my model to classify many classes. For this problem in particular the class is age. Each age ...
0
votes
1answer
20 views

How do we visualize data in hierarchical clustering?

Can anybody tell me how to do visualization when applying hierarchical clustering to data with more than 2 features? Do we need to do dimensionality reduction before each clustering?
1
vote
0answers
34 views

Is there a representation of the separating hyperplane in t-sne?

I have used t-sne to visualize a set of images which I have used for training a binary classifier. Let us assume that the binary classifier is trained to detect cat(1) vs. no-cat(0). I have used the ...
1
vote
3answers
51 views

K means visualisation after reducing dimensionality with PCA

In clustering ($K$ means, for example) when I have $N$ features and after creating the model (with this $N$ features) to visualize this model I need to reduce this $N$ dimensions into $2$ or $3$ ...
2
votes
1answer
63 views

How to reduce position changes after dimensionality reduction?

Disclaimer: I'm a machine learning beginner. I'm working on visualizing high dimensional data (text as tdidf vectors) into the 2D-space. My goal is to label/modify those data points and recomputing ...
0
votes
0answers
5 views

Deep Q Learning state dimensionality

How important is the dimensionality of each state for Deep Q Learning? I have a set of 15 unique playing cards from a deck of 52 playing cards. A given state is represented by the respective card ...