I am performing regression analysis on some data. I keep getting very high training score and low test score. My code is below, what can i do to enhance it? Thank you in advance.

# coding: utf-8

# In[1]:

#Importing modules
import sys
import math 
import itertools
import numpy as np
import pandas as pd
from numpy import genfromtxt
from matplotlib import style
import matplotlib.pyplot as plt
from sklearn import linear_model
from matplotlib import style, figure
from sklearn.linear_model import LassoCV
from sklearn.linear_model import RidgeCV
from sklearn.linear_model import LinearRegression
from sklearn.cross_validation import train_test_split

# In[2]:

#Importing data
df = np.genfromtxt('/Users/Studies/Machine_learning/reactivity/main_us.csv', delimiter=',')
#To skip the header ad skiprpws=0

# In[3]:

X = df[0:,1:306]
y = df[0:,0]

# In[4]:

print (X).shape
print (y).shape
display (X)
display (y)
print (y)

# In[5]:


# In[6]:

#Apply StandardScaler for feature scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform (X_test)
print len(X_test), len(y_test)

# In[7]:

#Applying PCA for dimnetionality reduction

from sklearn.decomposition import PCA
pca = PCA()
X_train = pca.fit_transform(X_train)
X_test = pca.transform(X_test)

#Checking shape after scaling
print ("Checking shape after scaling")
print (X_train.shape)
print (X_test.shape)


print ("Graph")
plt.scatter (X_train[:,0], X_train[:,1], c=y_train, edgecolor='none', alpha=0.5, cmap=plt.cm.get_cmap('rainbow',6))
plt.xlabel('Component 1')
plt.ylabel('Component 2')

print ('You are looking at a high dimentional data explained by 2 components')
print ('Eeven though these components hold some information, but this to seperate the components apart')


#Checking shape after scaling 
print (X_train.shape)
print (y_train.shape)
print (X_train.shape)

# In[8]:

alphas = 10**np.linspace(10,-2,100)*0.5

# In[9]:

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import Ridge, Lasso

for Model in [Ridge, Lasso]:
    model = Model()
    print('%s: %s' % (Model.__name__,
                      cross_val_score(model, X, y).mean()))

# Out[9]:

Ridge: -1.3841312374053019
Lasso: -1.164517926682712

# In[10]:

import numpy as np
from matplotlib import pyplot as plt

alphas = np.logspace(-3, -1, 30)

plt.figure(figsize=(5, 3))

for Model in [Lasso, Ridge]:
    scores = [cross_val_score(Model(alpha), X, y, cv=3).mean()
            for alpha in alphas]
    plt.plot(alphas, scores, label=Model.__name__)

plt.legend(loc='lower left')
plt.ylabel('cross validation score')

# In[11]:

# alpha = 0.1
model = Ridge(alpha = 0.1)
print model.score(X_train,y_train)   
print model.score(X_test,y_test)

# alpha = 0.01
model1 = Ridge(alpha = 0.01)
print model.score(X_train,y_train)   
print model.score(X_test,y_test)

# alpha = 0.001
model2 = Ridge(alpha = 0.001)
print model.score(X_train,y_train)   
print model.score(X_test,y_test)

# alpha = 0.0001
model3 = Ridge(alpha = 0.0001)
print model.score(X_train,y_train)   
print model.score(X_test,y_test)

# Out[11]:


# In[12]:

modelCV = RidgeCV(alphas = [0.1, 0.01, 0.001,0.0001], store_cv_values = True)
modelCV.alpha_  #giving 0.1
print modelCV.score(X_train,y_train)  # giving 0.36898424479812919 which is the same score as ridge regression with alpha = 0.1
print modelCV.score(X_test,y_test) 

# Out[12]:


1 Answer 1


I am not going to much of your code as i can see you have only imported all the libraries: You are facing with Over-fitting issue, The below are few of things when i come accross the same situation:

  1. Build multiple models and check Goodness of fit and then implement.
  2. Cross-validation is something which you should look into to make sure you have choosen the right model.

How to deal with it: 1. Train with more data:(It won’t work every time, but training with more data can help algorithms detect the signal better.) 2. Remove features.(because every variable will have variance so even if it is not significant it will try to explain the variance of an dependent variable during training but in test, it will fail because it is not significant enough) 3. Early stopping: (Early stopping refers stopping the training process before the learner passes that point.) 4. Regularization:(It is a way to get a stable model) 5. Ensembling:(My favorite) Ensembles are machine learning methods for combining predictions from multiple separate models. There are a few different methods for ensembling, but the two most common are:

Bagging attempts to reduce the chance of overfitting complex models.

It trains a large number of "strong" learners in parallel. A strong learner is a model that's relatively unconstrained. Bagging then combines all the strong learners together in order to "smooth out" their predictions. Boosting attempts to improve the predictive flexibility of simple models. It trains a large number of "weak" learners in sequence. A weak learner is a constrained model (i.e. you could limit the max depth of each decision tree). Each one in the sequence focuses on learning from the mistakes of the one before it. Boosting then combines all the weak learners into a single strong learner. While bagging and boosting are both ensemble methods, they approach the problem from opposite directions. Bagging uses complex base models and tries to "smooth out" their predictions while boosting uses simple base models and tries to "boost" their aggregate complexity.

  • $\begingroup$ Thank you. I already did LassoCV and RidgeCV. 1. I cant get more data, this is all the data i have 2. I already used PCA 3. not familiar with this, will check it 4. already implemented 5. Will try ensembling. Thank you. $\endgroup$ Commented Mar 6, 2019 at 9:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.