0
$\begingroup$

I am making a simple test on multiple linear regression.

  1. Importing datasets and libraries

    import numpy as np
    import matplotlib as plt
    import pandas as pd
    
    dataset = pd.read_csv('50_Startups.csv')
    X = dataset.iloc[:, :-1]
    y = dataset.iloc[:, 4]
    
  2. Split categorical features into numeric

    from sklearn.preprocessing import LabelEncoder, OneHotEncoder
    le = LabelEncoder()
    oe = OneHotEncoder(categorical_features=[3])
    X.iloc[:, 3] = le.fit_transform(X.iloc[:, 3])
    X = oe.fit_transform(X).toarray()
    
  3. Removing a variable to avoid dummy variable trap

    X = X[:, 1:]
    
  4. Building an optimal model using multiple linear regression and statsmodel:

    import statsmodels.api as sm
    X = np.append(arr = np.ones((50, 1)).astype(int), values = X, axis = 1)
    

To run the model, I made the following function:

def calculateMultReg(x, sl):
    dataSetLength = len(x[0])
    for i in range(0, dataSetLength):
        regressor_OLS = sm.OLS(endog=y, exog=X).fit()
        maxPValue = max(regressor_OLS._results.pvalues).astype(float)
        if(maxPValue>sl):
            for j in range(0, dataSetLength-i):
                if(maxPValue==regressor_OLS._results.pvalues[j]):
                    x = np.delete(x, j, 1)
        regressor_OLS.summary()
        return x
calculateMultReg(X, 0.05)

enter image description here

The result was that X was as the same as the initial one imported at the top of the script.

But when I do it manually (the multiple linear regression) using the following:

#First pvalue comparing. Column with index 2 has been removed.
X_opt = X[:, [0, 1, 3, 4, 5]]
#ordinary least square model
regressor_OLS = sm.OLS(endog=y, exog=X_opt).fit()
regressor_OLS.summary()

#Second pvalue comparing. Column with index 1 has been removed.
X_opt = X[:, [0, 3, 4, 5]]
#ordinary least square model
regressor_OLS = sm.OLS(endog=y, exog=X_opt).fit()
regressor_OLS.summary()
...

enter image description here

The final X_opt was only 2 fields.

So why this is happening? and how I test a test set?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.