1
$\begingroup$

Im trying to make regression with sklearn between one feature and one result. This is the dataset that I have:

       bruto  ukupno gradjevinski din
0    2494.98                857951.27
1    2912.60                694473.11
2    3397.50               1310529.72
3    2678.00                199688.14
4    4310.00               1377366.95
5    2086.28                569312.33
6    3061.80                660803.42
7    4095.00               1187732.61
8    3997.00               1304793.08
9    6503.88               1659629.13
10   6732.00               1264178.31
11    940.10                172497.94
12   1543.00                598772.40
13   5903.85                809681.19
14   2861.61                333983.85
15   3682.76               1430771.50
16   2802.00               1145812.21
17   3032.00                356840.54
18   2635.00                543912.80
19   3749.00               1004940.27
20   4300.50               1889560.55
21   9722.00               2137376.95
22   3823.33                891633.50
23   1648.21                335115.40
24  24575.00              19273129.14
25   3926.00               1223803.28
26   3228.00                874000.00
27   4062.00               1090000.00
28   1316.24                332718.54
29   2497.99                519398.70
30  12123.94               2504783.69
31   2057.50                957042.37
32   2495.00                857951.27
33   3770.73               1743978.85
34    864.00                251269.48
35    774.71                192487.26

I have found the correlation between feature and result with .corr():

                            bruto  ukupno gradjevinski din
bruto                    1.000000                 0.878914
ukupno gradjevinski din  0.878914                 1.000000

I have corr of 0.87 and I think that that is very decent for regression, but when I make regression model and when i get cross-val score, i get value for cross-val score thats negative and bigger then 1 (sometimes -50.23) and thats very strange to me. I have tried with a lot of different models and with different number of folds but the results are the same. This is the code for regression:

features = df[['bruto']]
results = df[['ukupno gradjevinski din']]

regressors = [["Linear Regression", LinearRegression(normalize=False)],
              ["Lasso Regression", Lasso(normalize=False)],
              ["Gaussian Process Regressor", GaussianProcessRegressor()],              
              ["SVR linear", SVR(kernel = 'linear', gamma='scale', max_iter = 1500)],
              ["SVR poly 2", SVR(kernel = 'poly', degree=2, gamma='scale', max_iter = 1500)],
              ["SVR poly 3", SVR(kernel = 'poly', degree=3, gamma='scale', max_iter = 1500)],
              ["SVR poly 4", SVR(kernel = 'poly', degree=4, gamma='scale', max_iter = 1500)],
              ["SVR poly 5", SVR(kernel = 'poly', degree=5, gamma='scale', max_iter = 1500)],
              ["SVR rbf C=0.01", SVR(kernel = 'rbf', C=0.01, gamma='scale', max_iter = 1500)],              
              ["SVR rbf C=0.1", SVR(kernel = 'rbf', C=0.1, gamma='scale', max_iter = 1500)],
              ["SVR rbf C=0.5", SVR(kernel = 'rbf', C=0.5, gamma='scale', max_iter = 1500)],
              ["SVR rbf C=1", SVR(kernel = 'rbf', C=1, gamma='scale', max_iter = 1500)],              
              ["SVR rbf C=10", SVR(kernel = 'rbf', C=10.0, gamma='scale', max_iter = 1500)],
              ["SVR rbf C=20", SVR(kernel = 'rbf', C=20.0, gamma='scale', max_iter = 1500)],
              ["SVR rbf C=50", SVR(kernel = 'rbf', C=50.0, gamma='scale', max_iter = 1500)],              
              ["SVR sigmoid", SVR(kernel = 'sigmoid', gamma='scale', max_iter = 1500)],
              ["GradientBoostingRegressor", GradientBoostingRegressor()],
              ["RandomForestRegressor", RandomForestRegressor(n_estimators = 150)],
              ["DecisionTreeRegressor", DecisionTreeRegressor(max_depth=10)],
              ["Bagging Regressor TREE", BaggingRegressor(base_estimator = DecisionTreeRegressor(max_depth=15))],
              ["Bagging Regressor FOREST", BaggingRegressor(base_estimator = RandomForestRegressor(n_estimators = 100))],
              ["Bagging Regressor linear", BaggingRegressor(base_estimator = LinearRegression(normalize=True))],
              ["Bagging Regressor lasso", BaggingRegressor(base_estimator = Lasso(normalize=True))],
              ["Bagging Regressor SVR rbf", BaggingRegressor(base_estimator = SVR(kernel = 'rbf', C=10.0, gamma='scale'))],
              ["Extra Trees Regressor", ExtraTreesRegressor(n_estimators = 150)],
              ["K-Neighbors Regressor 1", KNeighborsRegressor(n_neighbors=1)],
              ["K-Neighbors Regressor 2", KNeighborsRegressor(n_neighbors=2)],
              ["K-Neighbors Regressor 3", KNeighborsRegressor(n_neighbors=3)],
              ["AdaBoostRegressor", AdaBoostRegressor(base_estimator=None)],
              ["AdaBoostRegressor tree", AdaBoostRegressor(base_estimator=DecisionTreeRegressor(max_depth=15))],
              ["AdaBoostRegressor forest", AdaBoostRegressor(base_estimator=RandomForestRegressor(n_estimators = 100))],
              ["AdaBoostRegressor lin reg", AdaBoostRegressor(base_estimator=LinearRegression(normalize=True))],
              ["AdaBoostRegressor lasso", AdaBoostRegressor(base_estimator = Lasso(normalize=True))]]


for reg in regressors:

     try:

           scores = cross_val_score(reg[1], features, results, cv=5)
           scores = np.average(scores)
           print('cross val score', scores)
           print()

     except:
          continue

I have tried to scale my features with Normalizer, StandardScaler and MinMaxScaler but results are the same. Any help?

$\endgroup$
1
$\begingroup$

I was about to post my answer on the other forum, but it's migrated to here.

There are a few crucial things that you should keep in mind:

  1. It’s not who has the best algorithm that wins. It’s who has the most data. (Banko and Brill, 2001)

Bank and Brill in 2001 made a comparison among 4 different algorithms, they kept increasing the Training Set Size to millions and came up with the above-quoted conclusion. And your data is too few!

  1. Whenever you talk about Linear Models, just remember their enemy -- The Outliers. If you plot your data, you can see that clearly.

enter image description here

  1. cross_val_score returns the R^2 by default for almost any Linear Model (i.e Regressor). The best value of this metric = 1 (i.e. totally fit), or = 0 (i.e. horizontal line), or it can be negative (i.e. worse than a horizontal line). More info here. Next in the experiment I conducted, you'll see how the results are valid.

  2. An alternative model would be the Multi-layer Perceptron Regressor; with number of layers = 3, the model would map any complicated function.

  3. Cross-Validation would best serve if you have enough data. However in your case, the CV scores vary noticeably.

Please ponder the results of the following self-explanatory Experiment:

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score
from sklearn.neural_network import MLPRegressor
from scipy.stats import pearsonr
import numpy as np
import matplotlib.pyplot as plt

X = np.array([2494.98,2912.6,3397.5,2678,4310,2086.28,3061.8,4095,3997,
              6503.88,6732,940.1,1543,5903.85,2861.61,3682.76,2802,3032,
              2635,3749,4300.5,9722,3823.33,1648.21,24575,3926,3228,4062,1316.24,
              2497.99,12123.94,2057.5,2495,3770.73,864,774.71]).reshape(-1, 1)

y = np.array([857951.27,694473.11,1310529.72,199688.14,1377366.95,569312.33,660803.42,1187732.61,
          1304793.08,1659629.13,1264178.31,172497.94,598772.4,809681.19,333983.85,1430771.5,1145812.21,
          356840.54,543912.8,1004940.27,1889560.55,2137376.95,891633.5,335115.4,19273129.14,1223803.28,
          874000,1090000,332718.54,519398.7,2504783.69,957042.37,857951.27,1743978.85,251269.48,192487.26])

X_, y_ = zip(*sorted(zip(X, y)))
plt.plot(X_, y_, '-x')
plt.title("Plot of Dataset")
plt.show()

print("Linear Regression :: Before Removing An Outlier")
reg = LinearRegression()
print(np.average(cross_val_score(reg, X, y, cv=3)))

X, y = X_[:-1], y_[:-1]
plt.plot(X, y, '-x')
plt.title("Plot of Dataset After Removing Outlier")
plt.show()

print("Linear Regression :: After Removing An Outlier")
reg = LinearRegression()
print(np.average(cross_val_score(reg, np.array(X).reshape(-1, 1), y, cv=3)))

print("Multi-layer Perceptron Regressor :: The Effect of Mapping Complicated / Non-Linear Function")
mlp = MLPRegressor(hidden_layer_sizes=(16, 16, 16), random_state=2020, activation='identity', max_iter=1000)
print(np.average(cross_val_score(mlp, np.array(X).reshape(-1, 1), y, cv=3)))

RESULTS

This after removing only one extreme value (without further exploration nor doing any fancy work like utilizing any outliers detector). As you can see, there would be no single line that fits all points.

enter image description here

Linear Regression :: Before Removing An Outlier
Average CVs Score: -1.7085612243433703

Linear Regression :: After Removing An Outlier
Average CVs Score: -0.12386365189238795

Multi-layer Perceptron Regressor :: The Effect of Mapping Complicated / Non-Linear Function
Average CVs Score: 0.16131374234257037
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.