Linear regression incorrect prediction using Matlab

Question

In the plot below the red crossed line is the actual curve and the crossed blue line is the predicted curve. I am using least squares for linear prediction. I have used 1:79 examples in training and the remaining for testing. The test data points are never seen during training. What is my mistake? Why am I getting such a weird prediction? I want to see the sine curve as the predicted output which should be very close to the original data.

%generate some data
x=linspace(0,2*pi,100)';
y=sin(x); %response

X=x;
y=y;
% Convert matrix values to double
X = double(X(1:79));
y = double(y(1:79));

% Plot data
plot(X, y, 'rx', 'MarkerSize', 10);

m = length(y);
% Add ones column
X = [ones(m, 1) X];

% Gradient Descent with Normal Equation
theta = (pinv(X'*X))*X'*y

% Predict  from 80 till last sample
test_samples = x(80:end);
test_samples_val = [ones(length(test_samples),1) test_samples];

% Calculate predicted value
pred_value = test_samples_val * theta;

X = vertcat(X, test_samples_val);
regressionline = X*theta;


% Plot predicted value with blue cross
plot(test_samples, pred_value, 'bx', 'MarkerSize', 10);

Answer 1

You are calculating a linear regression in a nonlinear environment, which means you will need nonlinear exogenous variables.

You are calculating $Y=\beta_0+\beta_1x$ (a linear equation) when you need at least a polynominal equation. The following code calculated this equation: $Y=\beta_0+\beta_1x+\beta_2x^2+\beta_3x^3$ (maybe not the best equation but a working one)

%generate some data
x=linspace(0,2*pi,100)';
y=sin(x); %response

hold on
X=x;
y=y;
% Convert matrix values to double
X = double(X(1:79));
y = double(y(1:79));

% Plot data
plot(X, y, 'rx', 'MarkerSize', 10);

m = length(y);
% Add ones column
X = [ones(m, 1) X X.^2 X.^3];

% Gradient Descent with Normal Equation
theta = (pinv(X'*X))*X'*y;

% Predict  from 80 till last sample
test_samples = x(80:end);
test_samples_val = [ones(length(test_samples),1) test_samples test_samples.^2 ...
test_samples.^3];

% Calculate predicted value
pred_value = test_samples_val * theta;

X = vertcat(X, test_samples_val);
regressionline = X*theta;


% Plot predicted value with blue cross
plot(test_samples, pred_value, 'bx', 'MarkerSize', 10);

Answer 2

Short answer

You want to model a non-linear function by a line that is really not suppose to work.

---

Exception

Well, for small ranges you can indeed approximate a sine wave by a line, for example we know that

$$ sin(\theta) \approx \theta$$ if $\theta \approx 0$

This linerization is common when modeling physical systems.

---

Comments of the experiment

You model was trainned using the range $[0,4.85]$ (approximately), using linearly spaced values, but tested in the range $[4.85,6.14]$.

If you evaluate the derivative in those ranges you will see that there are 2 sign flips, most of the interval happens with negative derivative so is very likely for a regression algorithm to fit it with a negative-signed set of weights.

I would advise you use a different model for your problem or use a mode suitable interval.

---

Suggestion

1 - To overcome your interval problem, you could reduce the range to $[1.5,4.8]$ radians or any interval without zero-crossings in the derivative of your function.

2 - You can shuffle your data or create random points within a (realistic) interval for this application