logistic regression algorithm fails to work

I'm trying to code my own logistic regression algorithm using Andrew NG's machine learning using Octave. lectures. So what I did was make a csv file, the first row being some parameter and the second one being the result:

121,1
124,0
97,0
104,0
110,0
...

Overall there are only 24 examples, but I've chosen points such that some pattern can be followed.

Here is my code:

x = data(:, 1);

y = data(:, 2)
m = length(y);

#plot(x, y, 'rx', 'MarkerSize', 10);
#xlabel('IQ');
#ylabel('Pass/Fail');
#title('Logistic Regression');

x = [ones(size(x, 1), 1) x];
alpha = 0.00001;
i = 15000;

g = inline("1 ./ (1 + exp(-z))")

theta = zeros(size(x(1, :)))';
j = zeros(i, 1);

for num = 1:i
z = x * theta;
h = g(z);
j = (1./m) * ( -y' * log( h ) - ( 1 - y' ) * log ( 1 - h))
grad = 1./m * x' * (h - y);
theta = theta - alpha * grad;
end

However the output of the sigmoid function shows every value below 0.5... surely this has to be wrong. I've also tried with different learning rates and iterations, but to no avail. What is wrong with the code, or data?

Help would be appreciated.

• How balanced is your datasets (ratio of 0's to 1's)? If it is very unbalanced, and there is not a sufficient relationship to the input, predictions staying below 0.5 can easily happen.
– Paul
Jan 31 '18 at 18:16
• pastebin.com/FQgnSf57 they are equally balanced, 12 0s and 12 1s Jan 31 '18 at 18:32
• x = data(:, 1); y = data(:, 2) what is in the first column? Feb 4 '18 at 15:41
• Do you mean column '0'? If so Octave indices start from 1 (I forgot to mention I was using that, so I edited my post). The first column = my feature/parameter #1 and the second column is the output Feb 4 '18 at 15:47
• @david oh boy, you are using octave :D Feb 4 '18 at 15:50

Consider that you are doing vector operation, change your cost function to the following:

(1 / m) * sum(((-y) .* (log(h)) - ((1 - y) .* log((1-h)))));

grad = (1./m) * (x' * (h - y))

Although the latter is just for precedence reassuring.

Based on the discussion in the chat, although the code calculates the cost in a wrong way, the reason the cost does not decrease is that the data is not linearly separable. Logistic regression is a simple algorithm which classifies successfully linearly separable data. Take a look at here.

• Now the output of every value of the sigmoid function is above 0.5...(all approximately 0.54) Feb 4 '18 at 16:18
• @david does it learn? Feb 4 '18 at 16:19
• Consider that what I said was irrelevant to sigmoid. maybe stupid but instead of double quote use single quotation for your inline function. Feb 4 '18 at 16:20
• I mean the J value stays constant at approximately 0.69 after every iteration and never reduces so I don't think so. Feb 4 '18 at 16:22
• I tried using single quotes, no difference :( Feb 4 '18 at 16:22

In general, there's nothing wrong with the output of calling the sigmoid being below 0.5. All the sigmoid does is to "squash" the data into a tight range that is consistent with a probability value (which must be between $0$ and $1$) before the next step of fitting the model.

Now if you're saying that values fail to converge, or get stuck at some small value during iterations, then I would guess there is a bug in your implementation of the loss/cost/error function, its gradient and related quantities.

You should definitely update the matrix * operations to the element-wise .* in this line:

j = (1./m) * ( -y' * log( h ) - ( 1 - y' ) * log ( 1 - h))

Also, I'm a bit suspicious of the plus & minus signs in the following equations. You should double-check them against the original mathematical equations to make sure, as I don't think the signs look exactly right:

j = (1./m) * ( -y' * log( h ) - ( 1 - y' ) * log ( 1 - h))
grad = 1./m * x' * (h - y);