Curse of Dimensionality : How many dimensions is too many dimensions?

Question

Say I have a dataset with 1000 columns and 3M rows. I know that this is will definitely suffer from Curse of Dimensionality and that I need to reduce the number of dimensions. But to what extent am I supposed to reduce the dimensions by?

With each additional dimension the number of datapoints required such that the data is not too sparse increases exponentially according to my understanding.

So how do I know know for different number of columns what the golden number of data points is? Assuming that I have the capability to collect infinite amount of data but would still have a small cost associated for each datapoint, how much should I collect?

I am using these slides to understand the concept: http://www.dataminingbook.info/pmwiki.php/Main/BookPathUploads?action=download&upname=slides-chap6.pdf

Answer 1

As karthikeyan mg mention in his answer, you could use the explained variance score to get an idea of how many columns you can drop. Unfortunately, there isn't a magic number to know in advance. If you write code in Python, you should read this blog post in towardsdatascience.com

An Approach to Choosing the Number of Components in a Principal Component Analysis

After you rescale your data to 0-1, you can run this snippet of code and get a plot of variance loss for each components number you choose.

#Fitting the PCA algorithm with our Data
pca = PCA().fit(data_rescaled)
#Plotting the Cumulative Summation of the Explained Variance
plt.figure()
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('Number of Components')
plt.ylabel('Variance (%)') #for each component
plt.show()

This is the result from the blog post. As you can see, the variance starts to drop a lot after the 5 components. So, this number might be the one you could use.

Answer 2

You can use explained variance score from Sklearn which gives the score of explained variance vs dimension of data. In statistics, explained variation measures the proportion to which a mathematical model accounts for the variation (dispersion) of a given data set. More the variation in the model, less it is prone to overfit and vice versa.

Answer 3

I don't know if there is a method to know how much data you need, if you don't underfit, then usually the more the better. To reduce dimensionality use PCA, and this will tell you the amount of variance lost with every dimension less. And maybe this article is useful: https://towardsdatascience.com/predicting-the-effect-of-more-training-data-by-using-less-c3dde2f9ae48