Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I'm trying to study the effect of curse of dimensionality in classification algorithms.
I have a simple dataset with only 1 feature. So i can represent the set in a 1 dimension. If now i want to introduce some noise in this dataset, is it correct to add another feature with random values to my dataset ?
In addition to what Lupacante conceptually and nicely showed such that the added feature(s) has(have) to be informative for the model otherwise it can get ignored by majority of models (perhaps easily by regularized models), I would like add that you also increase the dimensionally of the feature space synthetically using many simple mathematical expressions as well. Concertedly, let's say your only feature (column) is:
x
You may easily construct other features like (it is common practice actually in physical sciences):
x$^2$, x$^3$, x$^{0.5}$, $sin(x)$, x$^2$sin(x),...
Till you hit the so-called the curse of dimensionality for your exercise. Although I am not yet sure what features/num_samples ratio exactly causes the curse of dimensionality! What I have gathered so far about the curse of dimensionality has been very subjective.
It depends on what is understood as noise, since a noise source can be interpreted as any way of corrupting/altering the data.
Technically, if you want to add noise to your dataset you can proceed as follows:
or
Adding noise is not the same as changing the dimension of the feature space. If the data is linearly separable in the original feature space, it will be also separable although you add an extra random feature. Take a look at figure 1 and figure 2. Figure depicts the scatter plot (var1_1 vs var1_1) of a linear separable data in a one dimensional feature space. Figure 2 depicts the scatter plot of the same feature space with an extra random feature, now the dimension is 2, but the data is still linearly separable. You only have to look at the projection of the data in the var1_1 axis.
Figure 1: Scatter plot of separable data in a 1-D feature space
Figure 2:Scatter plot of separable data in a 2-D feature space, which is the same as the previous space plus an extra random feature
[![Scatter plot of separable data in a 2-D feature space], which is the same as the previous space plus an extra random feature[2]](https://i.stack.imgur.com/mE9dw.png)
If you want to evaluate the robustness of your prediction model against noise, I will take option 1, since it not straightforward to derive what kind of noise to apply in the feature space. If you are working with images, you can blur them or if you are dealing with audio files, you can add white gaussian noise, or another kind of noise source, for example another mixing the original audio files with other sound sources.
The problem with adding an extra feature with random values is that, if it's uninformative (as it likely is, given that its values are all random), it might get ignored by your classifier.
For example:
from sklearn.linear_model import LinearRegression
from numpy import arange, random, array
x1 = arange(100)
y = 3 + 2*x1
LinearRegression().fit(array([x1, x2]).T, y).coef_
returns array([ 2.00000000e+00, -1.30768001e-15]), meaning that the coefficient of the new feature (the one with random values) was practically set to $0$.
What you could do is this:
x = arange(100)
y = 3 + 2*x + random.randn(100)
Then, if you try plotting y against x, you'll see that the values don't lie on a perfectly straight line, but rather they deviate from it slightly (and randomly).
