Questions tagged [dimensionality-reduction]
Dimensionality reduction refers to techniques for reducing many variables into a smaller number while keeping as much information as possible. One prominent method is [tag pca]
296
questions
1
vote
1
answer
41
views
Beginner basic clustering model and one-hot encoding?
I have a dataframe of natural disaster incidents in Afghanistan from 2016 - 2023.
Column names:
REGION (Northern, Eastern etc)
PROV_CODE (province)
PROV_NAME
DIST_CODE (district)
DIST_NAME
INC_DATE (...
- 13
0
votes
1
answer
28
views
Beginner clustering project, what are the input features and how do I analyze the data?
I am a beginner to data science. I have this dataset on natural disaster events in Afghanistan from 2016 - 2017. Columns:
REGION (ex. North, North West, etc)
PROVINCE_NAME (kind of like US 50 states)
...
- 13
0
votes
0
answers
13
views
How to deal with a small extra cluster in a tabular data?
I am working on a high dimensional tabular dataset with 1600 features and 9440 rows. No matter how I select the features, when I try to project my data into a 2d or 3d graph using dimensionality ...
0
votes
0
answers
12
views
Is there any advantage to providing multi-dimensional input to torch modules?
Most layer types in torch.nn such as torch.nn.Linear accept input with more than one dimension. Is there any advantage in doing so if you can shape your data to represent a certain arrangement in ...
- 1
0
votes
1
answer
33
views
How to explain the new features after a PCA?
Let's say I made a PCA in which I reduced from 10 dimensions to 3. And it clusters the classes correctly, but how do I explain which dimensions are better to predict? It is obvious that the 3 ...
0
votes
0
answers
10
views
What is the best sampling strategy for correlation analysis?
I have a big dataset, and i want to finds subspaces with high correlation among features. I want to take only samples of data. So, what is the best sampling strategy in this context.
Thanks
- 21
0
votes
0
answers
7
views
A query with regard to removing data from a dataset before clustering. - conceptual
I am in posession of data with regard to my domain which is energy economics. The dataset contains daily data on daily electricity demand along with the daily capacities of wind and solar plants for ...
0
votes
1
answer
60
views
Linear Model With Highly Correlated Attributes Producing Inconsistent Weights
I know that having correlated attributes violates the linear model assumption of independent attributes, and I'm not interested in creating a more sophisticated model to tease apart the dependent ...
- 3
1
vote
0
answers
32
views
Density distribution for feature analysis
I trained a ML model on original data with 6373 features, then I trained the same model on compressed data (using autoencoder) and I got an improvement. Finally, I trained the same model on reduced ...
0
votes
0
answers
21
views
Using Autoencoder in Python for Data Selection and Risk Level Calculation
I have a dataset that includes 100 data points from each sensor, representing various measurements. These measurements can be used to calculate the level of risk associated with each sensor. However, ...
- 1
0
votes
0
answers
11
views
Problem with 2 variable PCA loadings- Loadings are the same for all variables
I'm working on a problem for which i want to do some dimensionality reduction using 3 different PCAs of 2 variables each. Basically i want to perform a PCA and keep the first component between the ...
1
vote
0
answers
20
views
How to properly visualize high-dimensional embeddings along with the decision boundary in 2-D?
I have a number of embeddings (300-dimensional FastText vectors for each instance of each class) that I apply a classifier to (Logistic Regression for now). I want to visualize the embeddings as well ...
- 11
2
votes
1
answer
74
views
How to improve the preservation of the global data structure in UMAP?
I have a dataset, where the features are comprised of points arranged in a regular grid on a simplex. Each of these points are defined as follows: A point $\mathbf{x}$ on the simplex can be ...
- 121
0
votes
0
answers
21
views
Can TSNE and other visualisation methods separate multivariate normal blobs?
Consider we have two classes of points. Both of them come from a multivariate normal distribution with an unrestricted covariance matrix.
Let's assume, that the densities of those distributions do not ...
0
votes
0
answers
125
views
How to cluster multivariate time-series on different datasets?
I have following issue: I am trying to cluster similar countries with respect to different temporal features. Therefore, I have twelve different datasets each representing a different country. Each ...
- 1
0
votes
0
answers
172
views
Transformer-based autoencoder generates same output and bad embeddings
I am trying to implement the transformer-based autoencoder presented in this paper: https://arxiv.org/abs/2210.08288
The paper seems rather vague to me and I do not fully understand how the model is ...
0
votes
1
answer
32
views
important feature selection using dimensionality reduction algorithms
I have a dataset having more than 25000 features. I did perform noise removal using the histogram approach, and this dataset gets reduced to more than 5000 features. There are two classes, healthy and ...
0
votes
0
answers
34
views
Feature selection in high-dimensional datasets with sparse features
What are the most effective techniques for feature selection in high-dimensional datasets with sparse features in the field of natural language processing?
- 101
0
votes
0
answers
26
views
Why use feature-hashing instead of just remove random words from Bag-of-words?
As far as I understand Feature-Hashing ("Hashing Trick") is that we map some string to an index in an array e.g say we want the resulting dimension of our array to be 5, then maybe the text <...
- 460
0
votes
1
answer
211
views
How to efficiently reduce dimensions of one-hot encoded categorical values?
I'm currently working on a project where I'm using an LSTM to learn and predict sequences of categorical data.
My dataset consists of variable-length sequences of items $s_i = [x_{i_0}, x_{i_1}, ..., ...
- 13
-1
votes
1
answer
41
views
Determining "filters" dimension after a convolution operation
I tried to calculate the "filtered" dimension and I seem to be getting it wrong.
Below there is the image I am trying to calculate the "filtered" dimension for, where you have 192 ...
- 3
0
votes
0
answers
13
views
Larger Latent Space Dim for Point Cloud Autoencoder
So I'm trying to follow a paper that uses a AE to learn point clouds. The thing is, the dimension of the point cloud data is 3 (x, y, z), but the dimension of the latent space from what I can tell is ...
- 101
0
votes
0
answers
14
views
Preserving the frontier while reducing dimensionality
I have this problem: reduce the dimensionality of the data such that points belonging to the frontier of the convex hull in high dimension stay relatively close the frontier in low dimensions. I do ...
0
votes
0
answers
17
views
How to handle dimensionality differences over time or between subjects
Note: This question has in mind tabular data, rather than imaging/NLP.
In the situation of collecting data over long periods of time, instruments may change and collect more precise data. This leads ...
0
votes
0
answers
18
views
How to choose the optimal PCA kernel
In a chemometrics application, I need to reduce the dimensionality of a spectral scan. The standard PCA is linear. Not sure if the data is. How do I choose the most optimal PCA kernel?
0
votes
0
answers
191
views
Supervised UMAP on multi-label data
Is training a semi or supervised dimensionality reduced space with UMAP using multi-label targets supported & known to yield meaningful results (with respect to the unsupervised embedding)?
The ...
- 341
1
vote
1
answer
114
views
Clustering by using Locality sensitive hashing *after* Random projection
It is well known that Random Projection (RP) is tightly linked to Locality Sensitive Hashing (LSH). My goal is to cluster a large number of points lying in a d-dimensional Euclidean space, where $d$ ...
0
votes
0
answers
67
views
Is it possible to use text Auto Encoders without text generation?
I have a use case where I have large texts, and a lot of it. Pretty often the sequence length exceeds 1000 tokens. I need a lower dimensional compression of the texts as an input for a classifier. The ...
- 101
0
votes
0
answers
15
views
Reducing the dimensionality of entire dataset
I have a lot of datasets with different shapes. For example few of them are (90, 892), (74, 853), (93, 765), ... etc.
I want to convert this shape to (90, 4), (74, 4), (93, 4) ... (x, 4). And, after ...
- 1
1
vote
1
answer
103
views
How to describe this UMAP connectivity figure
I generated this UMAP connectivity diagram of my research data. How do I interpret/describe this plot regarding the UMAP connectivity?
Is it correct to say that:
As there is a lot of connectivity ...
- 147
0
votes
1
answer
49
views
Classification problem with no context in numerical features
I have an extremely abstract and numeric data with equally abstract objective.
I have around 3000 rows of train data (df_train), where I have a binary target ...
- 118
2
votes
0
answers
71
views
How to understand to what maximum size you can reduce the dimension of data and avoid the curse of the dimensionality?
i have a question, maybe someone could help me. I use t-sne (also tried umap) to reduce the dimensionality of the text embeddings dataset (size of embedding 300). after that I will cluster using ...
- 183
0
votes
0
answers
27
views
How can "profiling" be done in a dataset? What are the different techniques?
I am currently doing an analysis in which I need to "profile" each record. For example, let's say I have a dataset of accounts with customer information (name, id, address, money spent, ...
0
votes
0
answers
17
views
PCA, Better performances with 300 components rather than 400 components : why?
I am building this content based image retrieval system.
I basically extract feature maps of size 1024x1x1 using any backbone.
I then proceed to apply PCA on the extracted features in order to ...
1
vote
1
answer
99
views
What is the Purpose of Feature Selection
I have a small medical dataset (200 samples) that contains only 6 cases of the condition I am trying to predict using machine learning.
So far, the dataset is not proving useful for predicting the ...
- 407
0
votes
0
answers
39
views
How to Approach Linear Machine-Learning Model When Input Variables are Inconsistent
Disclaimer: I'm relatively new to the data science and ML world -- still trying to get a firm grasp on the fundamentals.
I'm trying to overcome a regression challenge involving a large, multi-...
0
votes
1
answer
87
views
An autoencoder setup for anomaly detection
I am doing anomaly detection using machine learning. i have tried different models like isolation forest, SVM and KNN. The maximum accuracy that I can get from each of them is $80\%$ accordind to my ...
- 11
-1
votes
2
answers
156
views
Why Do a Set of 3 Clusters Across 1 Dimension and a Set of 3 Clusters Across 2 Dimensions Form 9 Apparent Clusters in 3 Dimensions?
I am sorry if this is a well-known phenomenon but I can't quite wrap my head around this. I have a related question: How To Develop Cluster Models Where the Clusters Occur Along Subsets of Dimensions ...
1
vote
0
answers
37
views
Accuracy drops when adding a fully connected layer for dimensionality reduction to a ResNet50
I'm training a ResNet50 for image classification and I'm interested in decreasing the dimensionality of the embedded layer, in order to apply some clustering techniques.
The suggested dimension is ...
- 61
0
votes
0
answers
21
views
Single scalar from vector
I am aware that this question is very general, but I found this question and it made me curious. What are the sensible ways that you can think of to derive a single scalar value from a vector? Of ...
- 21
0
votes
1
answer
25
views
Dimensionality Reduction of Curved Structural Data
I have been using PCA dimensionality reduction on datasets that are quite linear and now I am tasked with the same on datasets that are largely curved in space. Imagine a noisy sine wave for ...
- 111
1
vote
1
answer
22
views
How to choose Recursive Feature Elimination parameters
in my project I have >900 features and I thought to use Recursive Feature Elimination algorithm to reduce the dimensionality of my problem (in order to improve the accuracy).
But I can't figure out ...
0
votes
0
answers
30
views
Searching machine learning algorithm for regression problem with many features
I have a machine learning problem with about 160 features and 400 cases and I want to find the best predictors for a continuous outcome. The dataset contains variables of psychotherapists and clients. ...
2
votes
1
answer
1k
views
What is the meaning of preserving local or global structure of the data?
I read about PaCMAP dimensionality reduction method (PaCMAP).
They wrote that this method preserving both local and global structure of the data in original space.
...
- 297
1
vote
2
answers
172
views
Using PCA for Dimensionality Expansion
I was trying to use t-SNE algorithm for dimensionality reduction and I know this was not the primary usage of this algorithm and not recommended. I saw an implementation here. I am not convinced about ...
- 13
2
votes
1
answer
356
views
Can one use PCA to reduce the dimensionality of One-Hot-Encoded data?
I read a couple times that PCA was used as a method to reduce dimensionality for one-hot-encoded data. However, there were also some comments that using PCA is not a good idea since one-hot-encoded ...
- 21
1
vote
0
answers
19
views
Nearest neighbor face recognition in eigenspace when using dot product of test set with eigenvectors does not match the performance when using sklearn
I am trying to perform Face recognition using PCA (eigenfaces). I have a set of N training images (of dimensions M=wxh), which I have pre-processed into a vertical ...
- 111
1
vote
0
answers
26
views
Understanding Linear Discriminant Analysis
I am trying to understand the results of a Linear Discriminant Analysis.
For that purpose I used the Iris dataset.
I plotted the two first features:
Then, I projected these two features in the new ...
1
vote
2
answers
240
views
Dimensionality reduction for millions of features
I have a dataset with 10 million observations and 1 million sparse features.
I would like to build a binary classifier for predicting a particular feature of interest.
My main problem is how to deal ...
- 11
2
votes
1
answer
2k
views
Can t-SNE be applied to visualize time series datasets
I have multiple time-series datasets containing 9 IMU sensor features. Suppose I use the sliding window method to split all these data into samples with the sequence length of 100, i.e. the dimension ...
- 23