2
$\begingroup$

I'm trying to study the effect of curse of dimensionality in classification algorithms.

I have a simple dataset with only 1 feature. So i can represent the set in a 1 dimension. If now i want to introduce some noise in this dataset, is it correct to add another feature with random values to my dataset ?

$\endgroup$
3
$\begingroup$

In addition to what Lupacante conceptually and nicely showed such that the added feature(s) has(have) to be informative for the model otherwise it can get ignored by majority of models (perhaps easily by regularized models), I would like add that you also increase the dimensionally of the feature space synthetically using many simple mathematical expressions as well. Concertedly, let's say your only feature (column) is:

x

You may easily construct other features like (it is common practice actually in physical sciences):

x$^2$, x$^3$, x$^{0.5}$, $sin(x)$, x$^2$sin(x),...

Till you hit the so-called the curse of dimensionality for your exercise. Although I am not yet sure what features/num_samples ratio exactly causes the curse of dimensionality! What I have gathered so far about the curse of dimensionality has been very subjective.

| improve this answer | |
$\endgroup$
3
$\begingroup$

It depends on what is understood as noise, since a noise source can be interpreted as any way of corrupting/altering the data.

Technically, if you want to add noise to your dataset you can proceed as follows:

  1. Add noise to the raw data, i.e, corrupt the raw data with some noise distribution and with certain signal to noise ratio,

or

  1. Add noise to the feature space, but keeping its dimension.

Adding noise is not the same as changing the dimension of the feature space. If the data is linearly separable in the original feature space, it will be also separable although you add an extra random feature. Take a look at figure 1 and figure 2. Figure depicts the scatter plot (var1_1 vs var1_1) of a linear separable data in a one dimensional feature space. Figure 2 depicts the scatter plot of the same feature space with an extra random feature, now the dimension is 2, but the data is still linearly separable. You only have to look at the projection of the data in the var1_1 axis.

Figure 1: Scatter plot of separable data in a 1-D feature space Scatter plot of separable data in a 1-D feature space

Figure 2:Scatter plot of separable data in a 2-D feature space, which is the same as the previous space plus an extra random feature [Scatter plot of separable data in a 2-D feature space], which is the same as the previous space plus an extra random feature[2]

If you want to evaluate the robustness of your prediction model against noise, I will take option 1, since it not straightforward to derive what kind of noise to apply in the feature space. If you are working with images, you can blur them or if you are dealing with audio files, you can add white gaussian noise, or another kind of noise source, for example another mixing the original audio files with other sound sources.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I have my dataset. In practice which are the steps to add noise to my data ? $\endgroup$ – Qwerto May 22 '18 at 18:49
  • $\begingroup$ What kind of data do you have? The first step is to decide what type of noise you'll add, since it is not the same working with audio files or some other records like height measurements, web servers traffic, etc. Once you've decided the noise/perturbations you'll include, the next step is to have a statistical model of them or some proper way of generating this noise, which reflects real noisy data. $\endgroup$ – ignatius May 23 '18 at 6:55
2
$\begingroup$

The problem with adding an extra feature with random values is that, if it's uninformative (as it likely is, given that its values are all random), it might get ignored by your classifier.

For example:

from sklearn.linear_model import LinearRegression
from numpy import arange, random, array

x1 = arange(100)
y = 3 + 2*x1
LinearRegression().fit(array([x1, x2]).T, y).coef_

returns array([ 2.00000000e+00, -1.30768001e-15]), meaning that the coefficient of the new feature (the one with random values) was practically set to $0$.

What you could do is this:

x = arange(100)
y = 3 + 2*x + random.randn(100)

Then, if you try plotting y against x, you'll see that the values don't lie on a perfectly straight line, but rather they deviate from it slightly (and randomly).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.