How to implement linear regression

Question

I am having difficulty achieving the same result as in sklearn while implementing linear regression model from scratch.
After adjusting the learning rate, I obtained an AUC of 0.694 for this binary classification task after two iterations. But using the same data and sklearn's LinearRegression model, the AUC is 84.5 . Is there any way to improve this model?

Here is the code.

def loss_function(feat_x_i, y_i, weight_0, weights_i, not_bias):
    # In case we are computing the weight of a feature (not the bias)
    if not_bias != 0:
        g = feat_x_i[not_bias-1]
    else:
        # In case we are computing the bias term (w_0)
        g = 1            
    return (np.matmul(np.asarray(feat_x_i), np.asarray(weights_i)) + weight_0 - y_i)*g    


# Train the model
learning_rate = 0.000001
# Weight of each feature, and +1 for the bias term (the first element)
# X_train.shape = (4000, 20)
update_weight = {i: [0] for i in range(len(X_train[0])+1)}

# When to stop ?? I am currently looking at the AUC output to stop
run_n_times = 20 
for i in range(run_n_times):
    # For each weight
    for w in list(update_weight.keys()):
        cost = 0
        most_recent_weight = [update_weight[j][-1] for j in range(1, len(update_weight))]
        bias = update_weight[0][-1]
        
        # For each example in the training data set
        for i in range(len(X_train)):
            cost += loss_function(X_train[i], y_train[i], bias, most_recent_weight, w)
        get_w = update_weight[w][-1] - learning_rate*cost
        update_weight[w].append(get_w)

    #Test the model
    final_weight = [update_weight[j][-1] for j in range(1, len(update_weight))]
    final_bias_term = update_weight[0][-1]

    predict = []
    for m in range(len(X_val)):
        y_pred = np.matmul(np.asarray(X_val[m]), np.asarray(final_weight)) + final_bias_term 
        predict.append(y_pred)

    # Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores.
    auc = roc_auc_score(y_val, predict)
    print(auc)
    if auc > 0.7:
        print('auc = %.3f' % auc)
        break
```

Answer 1

Well there are many different ways you can improve your code. For now, You can do this to improve the model:

1. Try reducing the learning rate to a value like (0.00001 or 0.0001). Your implementation uses a very small learning rate (0.000001), which may make it difficult for the model to converge to a good solution.
2. Add regularization: Looks like your model is currently overfitting to the training data, which is why it is not performing well.
3. You are only running the training loop for 20 iterations, which may not be enough for the model to converge
4. Use a different optimization algorithm. You can try using a more advanced optimization algorithm like Adam or RMSprop.
5. Scale the input data: You can use the StandardScaler class. This can help the optimization algorithm converge faster and can improve the performance of the model.

And lastly, applying the sigmoid function to the dot product of the feature vector and the weight vector can improve the performance of the model.

These are just few ways, you can give it a try actually.