Skip to main content

Questions tagged [dimensionality-reduction]

Dimensionality reduction refers to techniques for reducing many variables into a smaller number while keeping as much information as possible. One prominent method is [tag pca]

Filter by
Sorted by
Tagged with
5 votes
1 answer
506 views

Illustrating the dimensionality reduction done by a classification or regression model

Tl;DR: You can predict something, but how do you explain the prediction? EDIT: I have built a website that tries to answer this question with means of embedding / visually clustering data according ...
2 votes
1 answer
52 views

Dealing with high dimensionality datasets

I have data of dimensionality (25000, 100, 500) i.e. 25000 rows each consisting of a 2 dimensional 100 X 500 matrix. Currently I am only applying CNN for ...
0 votes
0 answers
15 views

About autoencoder's latent state regularity

Suppose we are dealing with the problem of dimensionality reduction of an input $\mathbf{x}\in\mathbb{R}^N$, by employing an autoencoder, as a composition of the encoder and decoder map $\mathbf{x} \...
0 votes
1 answer
96 views

Can I cluster an aggregated data-set (grouped by) and apply dimensionality reduction?

I have data of sales, however it is in the millions, about 500M rows. I aggregate this data by factors such as location, shoptype, country_of_shop, cardtype, and then the aggregated statistic is: ...
0 votes
1 answer
23 views

The latest approach for feature dimenesion reduction

I have a feature matrix with 1200 rows and 18930 columns. The matrix is sparse and the original paper has used a stacked denoising autoencoder for dimensionality reduction. Since I want to enhance the ...
0 votes
2 answers
60 views

Dimensionality reduction for feature extraction when missing some feature values

I have two questions: 1-Which method is appropriate for dimensionality reduction for feature extraction when missing some feature values? 2-Which textbook is the best source for the answer in (1)?
2 votes
1 answer
827 views

Can one use PCA to reduce the dimensionality of One-Hot-Encoded data?

I read a couple times that PCA was used as a method to reduce dimensionality for one-hot-encoded data. However, there were also some comments that using PCA is not a good idea since one-hot-encoded ...
0 votes
2 answers
126 views

Need suggestions on customer segmentation

I have been tasked with performing customer segmentation for a Business to business use case based on customer purchase history. Can experts provide me inputs on how do I proceed with customer ...
3 votes
1 answer
822 views

How to deal with disconnected components in isomap?

While creating a nearest neighbor graph for isomap, there is a possibility that the graph is disconnected. In this case finding graph distances between all pairs of points will not be possible. Are ...
4 votes
2 answers
128 views

How to reduce position changes after dimensionality reduction?

Disclaimer: I'm a machine learning beginner. I'm working on visualizing high dimensional data (text as tdidf vectors) into the 2D-space. My goal is to label/modify those data points and recomputing ...
1 vote
0 answers
78 views

Using UMAP on text data (euclidean distance on jaccard distance matrix)

I am checking the capabilities of the UMAP dimensionality reduction algorithm, I am not sure whether the approach I am using is valid and does not violate the rules/limitations of this algorithm. ...
2 votes
2 answers
144 views

How to leverage description data in multi-class classification (dimensionality reduction)

I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as ...
0 votes
1 answer
1k views

Low memory error while performing degree 2 polynomial regression on (3000*1835) sized array

I am working on a problem to predict the revenue, a film will generate. Some of the features available in the data set are json collection for the crew, cast which worked in the film. I applied ...
0 votes
1 answer
1k views

k-means clustering over columns not rows

I have a table with 100K+ rows and 100+ columns all numeric. Rather than using k-means to cluster rows together (and creating a new column category that labels each ...
2 votes
2 answers
2k views

How to perform feature selection with Categorical Variables and Continuous Target, provided that data is not normally distributed?

I am trying a use multi linear regression model to predict the salaries of employees. I have a total of 88 dependent features from which 19 are categorical and the rest are continuous. I have managed ...
1 vote
2 answers
172 views

How to split and train a model for data in biology

I am using gene expression data that are float numbers and want to train classifiers in view of binary classification. Since I am a novice in this field I have some questions: The first classifier I ...
2 votes
1 answer
955 views

Keras - Autoencoder different from Encoder + Decoder

I build a CNN 1d Autoencoder in Keras, following the advice in this SO question, where Encoder and Decoder are separated. My goal is to re-use the decoder, once the Autoencoder has been trained. The ...
0 votes
1 answer
205 views

PCA in visual Analytics

I m studying visual analytics and i have a theoretical question about this topic. My professor introduced this schema in him slide For connect data to visualisation. Some topic is very easy to ...
0 votes
1 answer
438 views

How to efficiently reduce dimensions of one-hot encoded categorical values?

I'm currently working on a project where I'm using an LSTM to learn and predict sequences of categorical data. My dataset consists of variable-length sequences of items $s_i = [x_{i_0}, x_{i_1}, ..., ...
0 votes
1 answer
220 views

Is t-SNE good at clustering instances with the same trend?

I have a dataset of time-series data with 50k examples and a length of 90, like the images showed below: I was wondering whether t-SNE or any type of dimensionality reduction could group the ...
0 votes
0 answers
26 views

Dimenson reduction from a cosine similarity matrix

I have a silly little question: I have 200 press articles (string), I vectorize these articles with an embedding model (sentence embedding), so I have 1024 values per article. I then have a 200 x 1024 ...
1 vote
1 answer
289 views

Dimensionality reduction to correlate large number of variables

So I have this dataset with about 750 variables (columns) and 50,000 rows of entries. I would like to reduce the dimensionality of the dataset to say 25-50-100 dimensions and then compute a ...
0 votes
1 answer
526 views

Extension of NMF to 3D

AFAIK, Non-Negative Matrix Factorization (NMF) is the procedure of looking for matrices $A$ and $B$ such that $$Data_{ik} = \sum_j A_{ij} B_{jk}$$ My data matrix is in fact 3D. I would like to fit ...
1 vote
2 answers
1k views

Which algorithm can be used to reduce dimension of multiple time series?

In my dataset, a data point is essentially a Time series of 6 feature over a year per month so in all, it results in 6*12=72 features. I need to find class outliers so I perform dimensionality ...
1 vote
1 answer
380 views

How to structure my data into features and targets for PCA on Big Data?

I want to apply the PCA algorithm from Scikit-Learn.(https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html ) At the part where I have to separate the features and the ...
2 votes
1 answer
215 views

How can I reduce the number of dimensions using a Clustering algorithm in a mixed dataset?

I am working with a mixed data set, corresponding to TV consumption data, with the aim of reducing the number of features to only those relevant to detect TV consumption patterns (or consumption ...
4 votes
2 answers
979 views

What can we learn from PCA on non linear data?

Suppose we have dataset with 10 features which are not linear: ...
5 votes
2 answers
3k views

Multidimensional scaling producing different results for different seeds

I took the data from here and wanted to play around with multidimensional scaling with this data. The data looks like this: In particular, I want to plot the cities in a 2D space, and see how much it ...
1 vote
1 answer
877 views

Decision trees and Curse of Dimensionality

Since decision tree algorithm splits the training dataset one feature at a time, how the heck is possibly that it suffers from curse of dimensionality ?
0 votes
1 answer
34 views

TSNE plots of random data subsets are vastly different but labels are still clearly separated - what conclusions can we draw about the dataset?

I scraped a dataset of match data in a video game and labeled them according to their outcome (0 for loss, 1 for win). I wanted to see if there was actually any inherent relationship between the ...
1 vote
1 answer
70 views

Beginner basic clustering model and one-hot encoding?

I have a dataframe of natural disaster incidents in Afghanistan from 2016 - 2023. Column names: REGION (Northern, Eastern etc) PROV_CODE (province) PROV_NAME DIST_CODE (district) DIST_NAME INC_DATE (...
0 votes
1 answer
53 views

Beginner clustering project, what are the input features and how do I analyze the data?

I am a beginner to data science. I have this dataset on natural disaster events in Afghanistan from 2016 - 2017. Columns: REGION (ex. North, North West, etc) PROVINCE_NAME (kind of like US 50 states) ...
0 votes
0 answers
15 views

How to deal with a small extra cluster in a tabular data?

I am working on a high dimensional tabular dataset with 1600 features and 9440 rows. No matter how I select the features, when I try to project my data into a 2d or 3d graph using dimensionality ...
0 votes
0 answers
16 views

Is there any advantage to providing multi-dimensional input to torch modules?

Most layer types in torch.nn such as torch.nn.Linear accept input with more than one dimension. Is there any advantage in doing so if you can shape your data to represent a certain arrangement in ...
0 votes
1 answer
135 views

How to explain the new features after a PCA?

Let's say I made a PCA in which I reduced from 10 dimensions to 3. And it clusters the classes correctly, but how do I explain which dimensions are better to predict? It is obvious that the 3 ...
4 votes
2 answers
1k views

Can I do incremental learning with the sklearn implementation of Linear Discriminant Analysis

I have a large number of pictures that I would like to use LDA on. However, it requires too much memory, so I was wondering if it would be possible to make the learning incremental, using a sklearn ...
0 votes
0 answers
10 views

What is the best sampling strategy for correlation analysis?

I have a big dataset, and i want to finds subspaces with high correlation among features. I want to take only samples of data. So, what is the best sampling strategy in this context. Thanks
0 votes
0 answers
8 views

A query with regard to removing data from a dataset before clustering. - conceptual

I am in posession of data with regard to my domain which is energy economics. The dataset contains daily data on daily electricity demand along with the daily capacities of wind and solar plants for ...
4 votes
1 answer
3k views

What are the differences among Proper Orthogonal Decomposition (POD), Singular value decomposition (SVD) and principal component analysis (PCA)?

Proper Orthogonal Decomposition (POD), singular value decomposition (SVD), and principal component analysis (PCA) are three eigenvalue methods used to reduce a high-dimensional data set into fewer ...
0 votes
1 answer
248 views

Guidance needed with dimension reduction for clustering - some numerical, lots of categorical data

I've my data in a Pandas df with 25.000 rows and 1.500 columns without any NaNs. Of the columns about 30 contain numerical data which I standardized with StandardScaler(). The rest are cols with ...
0 votes
1 answer
82 views

Linear Model With Highly Correlated Attributes Producing Inconsistent Weights

I know that having correlated attributes violates the linear model assumption of independent attributes, and I'm not interested in creating a more sophisticated model to tease apart the dependent ...
0 votes
1 answer
128 views

An autoencoder setup for anomaly detection

I am doing anomaly detection using machine learning. i have tried different models like isolation forest, SVM and KNN. The maximum accuracy that I can get from each of them is $80\%$ accordind to my ...
10 votes
4 answers
4k views

What does the long curve-shape t-SNE mean?

I use 1-D CNN input 1*512 size time series data which randomly fragment segment, the output will classify input into 10 classes. After training the CNN, I apply t-SNE to the prediction which I fed in ...
1 vote
0 answers
33 views

Density distribution for feature analysis

I trained a ML model on original data with 6373 features, then I trained the same model on compressed data (using autoencoder) and I got an improvement. Finally, I trained the same model on reduced ...
1 vote
0 answers
22 views

How to properly visualize high-dimensional embeddings along with the decision boundary in 2-D?

I have a number of embeddings (300-dimensional FastText vectors for each instance of each class) that I apply a classifier to (Logistic Regression for now). I want to visualize the embeddings as well ...
2 votes
1 answer
98 views

How to improve the preservation of the global data structure in UMAP?

I have a dataset, where the features are comprised of points arranged in a regular grid on a simplex. Each of these points are defined as follows: A point $\mathbf{x}$ on the simplex can be ...
10 votes
2 answers
13k views

Reducing the dimensionality of word embeddings

I trained word embeddings with 300 dimensions. Now, I would like to have word embeddings with 50 dimensions: is it better to retrain the word embeddings with 50 dimensions, or can I use some ...
0 votes
0 answers
28 views

Can TSNE and other visualisation methods separate multivariate normal blobs?

Consider we have two classes of points. Both of them come from a multivariate normal distribution with an unrestricted covariance matrix. Let's assume, that the densities of those distributions do not ...
0 votes
1 answer
44 views

important feature selection using dimensionality reduction algorithms

I have a dataset having more than 25000 features. I did perform noise removal using the histogram approach, and this dataset gets reduced to more than 5000 features. There are two classes, healthy and ...
2 votes
3 answers
2k views

PCA for complex-valued data

I'm quite shocked for encountering this error on PCA from sklearn ValueError: Complex data not supported After trying to fit ...

1
2 3 4 5 6