1
$\begingroup$

Say I have a dataset with 1000 columns and 3M rows. I know that this is will definitely suffer from Curse of Dimensionality and that I need to reduce the number of dimensions. But to what extent am I supposed to reduce the dimensions by?

With each additional dimension the number of datapoints required such that the data is not too sparse increases exponentially according to my understanding.

So how do I know know for different number of columns what the golden number of data points is? Assuming that I have the capability to collect infinite amount of data but would still have a small cost associated for each datapoint, how much should I collect?

I am using these slides to understand the concept: http://www.dataminingbook.info/pmwiki.php/Main/BookPathUploads?action=download&upname=slides-chap6.pdf

$\endgroup$
1
$\begingroup$

As karthikeyan mg mention in his answer, you could use the explained variance score to get an idea of how many columns you can drop. Unfortunately, there isn't a magic number to know in advance. If you write code in Python, you should read this blog post in towardsdatascience.com

An Approach to Choosing the Number of Components in a Principal Component Analysis

After you rescale your data to 0-1, you can run this snippet of code and get a plot of variance loss for each components number you choose.

#Fitting the PCA algorithm with our Data
pca = PCA().fit(data_rescaled)
#Plotting the Cumulative Summation of the Explained Variance
plt.figure()
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('Number of Components')
plt.ylabel('Variance (%)') #for each component
plt.show()

This is the result from the blog post. As you can see, the variance starts to drop a lot after the 5 components. So, this number might be the one you could use.

enter image description here

| improve this answer | |
$\endgroup$
0
$\begingroup$

You can use explained variance score from Sklearn which gives the score of explained variance vs dimension of data. In statistics, explained variation measures the proportion to which a mathematical model accounts for the variation (dispersion) of a given data set. More the variation in the model, less it is prone to overfit and vice versa.

| improve this answer | |
$\endgroup$
  • $\begingroup$ According to what i understand the metric you have given seems to be similar to R^2. I am more interested in how the (ratio of required rows / no of columns) changes as no of columns increases assuming that most of the columns explain a non-negligible amount of variance. $\endgroup$ – Umesh Sai Gurram Mar 15 '19 at 7:35
0
$\begingroup$

I don't know if there is a method to know how much data you need, if you don't underfit, then usually the more the better. To reduce dimensionality use PCA, and this will tell you the amount of variance lost with every dimension less. And maybe this article is useful: https://towardsdatascience.com/predicting-the-effect-of-more-training-data-by-using-less-c3dde2f9ae48

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.