1
$\begingroup$

I have manually created a random data set around some mean value and I have tried to use gradient descent linear regression to predict this simple mean value.

I have done exactly like in the manual and for some reason my predictor coefficients are going to infinity, even though it worked for another case.

Why, in this case, can it not predict a simple 1.4 value?

clear all;
n=10000;
t=1.4;
sigma_R = t*0.001;
min_value_t = t-sigma_R;
max_value_t = t+sigma_R;
y_data = min_value_t + (max_value_t - min_value_t) * rand(n,1);
x_data=[1:10000]';


m=0
c=0
L=0.0001
epochs=1000 %iterations


for i=1:epochs
   y_pred=m.*x_data+c;
   D_m=(-2/n)*sum(x_data.*(y_data-y_pred));
   D_c=(-2/n)*sum((y_data-y_pred));
   m=m-L*D_m;
   c=c-L*D_c;
end
plot(x_data,y_data,'.')
hold on;
grid;
plot(x_data,y_pred)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%

question update: Hello , I have tried to write down your code in the Matlab language i am more femiliar. my feature matrices of the form NX2 [1,X_data] is called Xmat. i followed every step in converting the code, and i get in both Theta NAN. Where did i go wrong?

$ %%start Matlab code n=1000; t=1.4; sigma_R = t*0.001; min_value_t = t-sigma_R; max_value_t = t+sigma_R; y_data = min_value_t + (max_value_t - min_value_t) * rand(n,1); x_data=[1:1000]; L=0.0001; %learning rate %plot(x_data,y_data); itter=1000;

theta_0=0; theta_1=0; theta=[theta_0;theta_1];

itter=1000; for i=1:itter onss=ones(1,1000); x_mat=[onss;x_data]'; pred=x_mat*theta; residuals = (pred-y_data); for k=1:2 %start theta loop partial=2*dot(residuals,x_mat(:,k)); theta(k)=theta(k)-L*partial; end%end theta loop end % end itteration loop %%end matlab code $

$\endgroup$
  • $\begingroup$ Can add the plot of this random data points of yours x vs. y? How does it look like? This helps getting an answer. For example, is the x vs. y a convex-like function? $\endgroup$ – TwinPenguins Apr 11 at 15:50
  • $\begingroup$ When i used matlabs regress it wokred like a charm i took 1.4 and made random ofsset o.1 around it so it should return 1.4 $\endgroup$ – user94586 Apr 11 at 18:44
0
$\begingroup$

Although this answer shows code with Python, the theory is exactly the same. Two advices when running a linear regression via gradient descent from scratch:

  • use matrix notation for your features data (i.e. input data) so you can apply it to n-dimensional datasets (not just 1-D as in this case)
  • standardize your feature matrix data (although in this case you have actually 1-D, but it is a good practice which always works)

Your data plotted with matplotlib:

import numpy as np 
from numpy.random import randn, seed

x_data = 20 * randn(1000) + 100
y_data = x_data + (10 * randn(1000) + 50)
n=1000
t=1.4
sigma_R = t*0.001
min_value_t = t-sigma_R
max_value_t = t+sigma_R
y_data = min_value_t + (max_value_t - min_value_t) * np.random.rand(n,1)
x_data=np.array(range(1000))

pyplot.scatter(x_data, y_data)
pyplot.show()

enter image description here

Build your dataset to train the model:

final_df_DICT = {'X': x_data}
import pandas as pd
H = pd.DataFrame(final_df_DICT)
feature_matrix = np.zeros(n*2) 
feature_matrix.shape = (n, 2) 
feature_matrix[:,0] = 1 
feature_matrix[:,1] = H['X'] 
#standardize features
feature_matrix = (feature_matrix - feature_matrix.mean()) / feature_matrix.std()
target_data = y_data.reshape(len(y_data), )

Look at the feature matrix, having the first column filled with 1's and the second column having the actual x_data: enter image description here

Model training with your hiperparameters:

w = [0, 0]
L=0.0001
epochs=1000
iteration = 0
cost=[]
while iteration < epochs:
    pred = np.dot(feature_matrix, w)
    residuals = pred-target_data
    #we calculate the gradient for the 2 coeffs with the scalar product 
    for i in range(len(w)):
        partial = 2*np.dot(residuals, feature_matrix[:, i])
        w[i] = w[i] - L*partial

    iteration += 1
    computed_cost = np.sum(np.power((pred - target_data), 2)) / n

    cost.append(computed_cost)

print('coef: {}'.format(w))
print('cost: {}'.format(cost[-1]))

Result:

coef: [-1.80963253e+00 -6.15189807e-06] cost: 6.466287828899486e-07

Let's plot the fitted regression model predictions over the original dataset (we trained on the whole dataset, not taking into account any validation set in this case...):

my_predictions = np.dot(feature_matrix, w)
pyplot.scatter(feature_matrix[:, 1], target_data)
pyplot.scatter(feature_matrix[:, 1], my_predictions, color='r')

pyplot.show()

enter image description here

| improve this answer | |
$\endgroup$
  • $\begingroup$ Hello, why partial formula is the same for both m and c. our predictor is y=mx+c so dy/dm=x dy/dc=1 so we have different partials so why partial is the same for both m,c? Thanks $\endgroup$ – user94586 Apr 13 at 13:36
  • $\begingroup$ Nice question, and that is the key to make the calculus with matrix algebra: if you look at how the feature_matrix is created (i.e. the matrix containing your input data, x_data) you can see 2 columns: the first one is filled with 1's (this is for the parameter c) and the second column actually has the x_data (see image attrached). This way, you can use the same expression for several attributes, including the c parameter $\endgroup$ – German C M Apr 13 at 13:45
  • $\begingroup$ Hello i have tried to make a MATLAB version for the your code. It gives me NAN Where did i go wrong? t=1.2; n=1000; sigma_R = t*0.001; min_value_t = t-sigma_R; max_value_t = t+sigma_R; y = min_value_t + (max_value_t - min_value_t) * rand(n,1); x = [1:1000].'; m = length(y); x = [ ones(m,1) x]; n = size(x,2); theta_vec = [0 0]'; alpha = 0.002; err = [0 0]'; for kk = 1:10000 h_theta = (xtheta_vec); h_theta_v = h_thetaones(1,n); y_v = yones(1,n); theta_vec = theta_vec - alpha*1/msum((h_theta_v - y_v).*x).'; err(:,kk) = 1/m*sum((h_theta_v - y_v).*x).'; end $\endgroup$ – user94586 Apr 14 at 7:18
  • $\begingroup$ from my previos comment the only thing missing is feature_matrix but you said its not nesserary because X is [NX2] is [1 x_data] is it nesserary? $\endgroup$ – user94586 Apr 14 at 7:27
  • $\begingroup$ quick reply about your latest question: indeed it is not necessary to code it as a matrix, but think of the case when you have multiple attributes, anyway for your case you can do the weight update separately as you did; about the nan values, have you also tried to standardize your input values? $\endgroup$ – German C M Apr 14 at 8:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.