# How to implement linear regression

I am having difficulty achieving the same result as in sklearn while implementing linear regression model from scratch.
After adjusting the learning rate, I obtained an AUC of 0.694 for this binary classification task after two iterations. But using the same data and sklearn's LinearRegression model, the AUC is 84.5 . Is there any way to improve this model?

Here is the code.

def loss_function(feat_x_i, y_i, weight_0, weights_i, not_bias):
# In case we are computing the weight of a feature (not the bias)
if not_bias != 0:
g = feat_x_i[not_bias-1]
else:
# In case we are computing the bias term (w_0)
g = 1
return (np.matmul(np.asarray(feat_x_i), np.asarray(weights_i)) + weight_0 - y_i)*g

# Train the model
learning_rate = 0.000001
# Weight of each feature, and +1 for the bias term (the first element)
# X_train.shape = (4000, 20)
update_weight = {i: [0] for i in range(len(X_train[0])+1)}

# When to stop ?? I am currently looking at the AUC output to stop
run_n_times = 20
for i in range(run_n_times):
# For each weight
for w in list(update_weight.keys()):
cost = 0
most_recent_weight = [update_weight[j][-1] for j in range(1, len(update_weight))]
bias = update_weight[0][-1]

# For each example in the training data set
for i in range(len(X_train)):
cost += loss_function(X_train[i], y_train[i], bias, most_recent_weight, w)
get_w = update_weight[w][-1] - learning_rate*cost
update_weight[w].append(get_w)

#Test the model
final_weight = [update_weight[j][-1] for j in range(1, len(update_weight))]
final_bias_term = update_weight[0][-1]

predict = []
for m in range(len(X_val)):
y_pred = np.matmul(np.asarray(X_val[m]), np.asarray(final_weight)) + final_bias_term
predict.append(y_pred)

# Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores.
auc = roc_auc_score(y_val, predict)
print(auc)
if auc > 0.7:
print('auc = %.3f' % auc)
break
$$$$
`
• Your learning rate is currently set to 0.000001, which is very small. This means that your model is making very small updates to the weights at each iteration, which can slow down convergence. Try reducing the learning rate to a smaller value (e.g. 0.00001 or 0.0001)
– Vic
Dec 10, 2022 at 5:52