I'm making a multiclassifier model with 5 classes. (it is not important in my question whether it has 2 classes or 5 classes, though).
class distribution is very imbalanced.
So, I did resampling for oversampling to match each number of classes.
MY question is. Is there any statistical error or problem if I resample classes for oversampling(bootstrap resampling or SMOTE)?
can anyone explain for this approach?
Thanks in advance.
This depends on the data in your minority classes.
The data in each class can be considered as a sample of observations from some population. This sample may or may not represent well the whole population of instances from that class.
If the sample represents the population well, then oversampling will introduce only a small error. However, if the sample does not represent the population well, then oversampling will produce data that have statistical properties different from those of the population.
All estimates (like confidence intervals, prediction intervals) are calculated from the statistical properties of the sample (mean, variance, etc), the exact calculations being different for different distributions and learning algorithms. If statistical properties of your oversampled data are different from the statistical properties of their populations, you will get wrong estimates of the confidence and prediction intervals for your model.
I will illustrate this with an example.
Let's assume that you have 2-dimensional data (two features in each observaton) that belong to 2 classes. Data in each class are normally distributed with the standard deviation for each feature = 1. The population mean of the class 1 is (-2, 0). The population mean of the class 2 is (1, 0).
I illustrate a large population, taking 500 points for each class. The classes can be separated by logistic regression line as follows:
The regression line is almost exactly between the population means and is almost vertical because both ordinates of the population means are zero. If I take a couple of thousand points then it will be vertical.
The code for this picture:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
n_points1 = 500
n_points2 = 500
mu1 = np.array([-2,0])
mu2 = np.array([1,0])
sig = 1
cl1 = sig * np.random.randn(n_points1,2) + mu1
cl2 = sig * np.random.randn(n_points2,2) + mu2
print('Red points:')
print('mu_x = %.3f, mu_y = %.3f, std_x = %.3f, std_y = %.3f'
% (np.mean(cl2[:,0]), np.mean(cl2[:,1]), np.std(cl2[:,0]), np.std(cl2[:,1])) )
y1 = np.zeros((cl1.shape[0],1))
y2 = np.ones((cl2.shape[0],1))
X = np.vstack((cl1,cl2))
y = np.vstack((y1,y2))
logreg = LogisticRegression()
logreg.fit(X, y.ravel())
w = logreg.coef_[0]
a = -w[0] / w[1]
xx = np.linspace(-4,4)
yy = a * xx - (logreg.intercept_[0]) / w[1]
plt.scatter(cl1[:,0], cl1[:,1], s = 50, marker='.', color='green')
plt.scatter(cl2[:,0], cl2[:,1], s = 50, marker='.', color='red')
plt.plot(xx, yy, 'b-')
plt.xlim((-5, 4))
plt.ylim((-5, 5))
plt.grid()
plt.show()
Now, let's assume that the red class is under-represented (a minority class). I take 500 points for the green class, and 10 points for the red class. Then I oversample the red data by two methods. One is duplicating them 50 times, like bootstrap resampling (I color them red), and another is something like SMOTE (they are magenta). This is not exactly SMOTE. I simply added data that are in the middle between red data points. I was too lazy to calculate nearest neighbors for each observation in this simple example but it illustrates SMOTE nevertheless because in SMOTE, synthetic examples are generated inside the convex hull of existing minority examples which reduces the variance of the data.
This is what I get:
The blue line is for bootstrap, and the black line is for SMOTE. The sample means and standard deviations for the minority class are as follows:
Red points:
mu_x = 1.600, mu_y = -0.182, std_x = 0.719, std_y = 0.753
Magenta points:
mu_x = 1.625, mu_y = -0.174, std_x = 0.513, std_y = 0.550
The x-component of the sample mean is overestimated (it should be = 1) and standard deviations are underestimated (they should be = 1). As a result, the line separating the classes is different from the true line on the previous picture, so a lot of new data will be classified incorrectly.
Let's take a couple of more samples randomly from the same population. Again, the size of the minority class is 10, and I resample them using the same two methods.
Red points:
mu_x = 0.944, mu_y = 0.289, std_x = 0.943, std_y = 0.867
Magenta points:
mu_x = 0.974, mu_y = 0.298, std_x = 0.700, std_y = 0.617
This time x-component of the sample mean is OK, its y-component is a little overestimated, and the standard deviation is underestimated. The separating line is again incorrect.
Red points:
mu_x = 0.922, mu_y = -0.236, std_x = 1.066, std_y = 1.097
Magenta points:
mu_x = 0.924, mu_y = -0.244, std_x = 0.757, std_y = 0.749
In the last picture we are lucky to get a sample that has almost the same mean and standard deviation as the population. Therefore, the separating line is very close to the line in the first picture where the data were balanced. Notice that the standard deviation for SMOTE is always smaller because new data are added between the existing data, not outside of them.
You might consider undersampling instead of oversampling. Check this link.
The code for the last 3 pictures:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
n_points1 = 500
n_points2 = 10
mu1 = np.array([-2,0])
mu2 = np.array([1,0])
sig = 1
cl1 = sig * np.random.randn(n_points1,2) + mu1
cl2 = sig * np.random.randn(n_points2,2) + mu2
cl2 = np.tile(cl2,(n_points1//n_points2,1)) # oversampling
# A kind of SMOTE but not quite
cl3 = np.zeros_like(cl1)
for k in range(len(cl3)):
i = np.random.randint(low=0, high=n_points2, size=2)
cl3[k,:] = (cl2[i[0]]+cl2[i[1]])/2
print('Red points:')
print('mu_x = %.3f, mu_y = %.3f, std_x = %.3f, std_y = %.3f'
% (np.mean(cl2[:,0]), np.mean(cl2[:,1]), np.std(cl2[:,0]), np.std(cl2[:,1])) )
print('Magenta points:')
print('mu_x = %.3f, mu_y = %.3f, std_x = %.3f, std_y = %.3f'
% (np.mean(cl3[:,0]), np.mean(cl3[:,1]), np.std(cl3[:,0]), np.std(cl3[:,1])) )
y1 = np.zeros((cl1.shape[0],1))
y2 = np.ones((cl2.shape[0],1))
X = np.vstack((cl1,cl2))
X3 = np.vstack((cl1,cl3))
y = np.vstack((y1,y2))
logreg = LogisticRegression()
logreg.fit(X, y.ravel())
w = logreg.coef_[0]
a = -w[0] / w[1]
xx = np.linspace(-4,4)
yy = a * xx - (logreg.intercept_[0]) / w[1]
plt.scatter(cl1[:,0], cl1[:,1], s = 50, marker='.', color='green')
plt.scatter(cl2[:,0], cl2[:,1], s = 50, marker='.', color='red')
plt.plot(xx, yy, 'b-')
logreg.fit(X3, y.ravel())
w = logreg.coef_[0]
a = -w[0] / w[1]
yy = a * xx - (logreg.intercept_[0]) / w[1]
plt.scatter(cl3[:,0], cl3[:,1], s = 10, marker='.', color='magenta')
plt.plot(xx, yy, 'k-')
plt.xlim((-5, 4))
plt.ylim((-5, 5))
plt.grid()
plt.show()
