# minimization with a negative cost function: works in MATLAB, not in Python

I'm trying to use a particular cost function (based on doubling rate of wealth) for a classification problem, and the solution works well in MATLAB. See https://github.com/acmyers/compareCostFXs

When I try to do this in Python 2.7.6 I don't get any errors, but it only returns zeros for the theta values.

Here is the cost function and optimization method I've used in Python:

def costFunctionDRW(theta, X, y):

# Initialize useful values
m = len(y)
# Marginal probability of acceptance
marg_pA = sum(y)/m
# Marginal probability of rejection
marg_pR = 1 - marg_pA

# =============================================================
pred = sigmoid(np.dot(X,theta))
final_wealth_individual = (pred/marg_pA)*y + ((1-pred)/marg_pR)*(1-y)
final_wealth = np.prod(final_wealth_individual)
final_wealth = -final_wealth

return final_wealth

result = scipy.optimize.fmin(costFunctionDRW, x0=initial_theta, \
args=(X_array, y_array), maxiter=1000, disp=False, full_output=True )


Any advice would be much appreciated!

• What type of data is y? Is it an int or a float? Do you use from __future__ import division? – TheBlackCat Feb 5 '15 at 9:36
• y is an int... changed it to a float and solved the issue! thanks a ton! – acmyers Feb 6 '15 at 7:14
• Great! I have added the answer as an answer below, please mark it as correct so others can find it more easily. – TheBlackCat Feb 6 '15 at 9:42

The problem is with division on python 2.x. In python 2.x, division involving two integers produces an integer result. So 1/2==0. Python 3.x does not have this problem, 1/2==.5.

There are two ways to avoid this. First, you can always convert one value you a float. So 1./2==0.5 and 1/2.==0.5. However, you have to remember to do this everywhere, and if you forget it can lead to hard-to-find errors.

The more reliable method is to always put this at the top of your code: from __future__ import division. This will switch python 2.x to the python 3 behavior, so 1/2==.5. In python 3.x it does nothing, so it also makes your code python 3.x compatible in this regard.

If your number of cases is large, you may be running into a problem of numerical underflow in the line

final_wealth = np.prod(final_wealth_individual)

If each value of final_wealth_individual is between 0 and 1, multiplying them all together can lead to a result that is too small to represent as a floating point number, resulting in a value of 0.

To address this issue, take the log of final_wealth_individual and add them together instead of multiplying. Note that this will cause final_wealth to be negative, so you will not need to multiply it by -1 as you are currently.

• No luck with this solution. I'm also only using 100 examples/cases. – acmyers Feb 4 '15 at 6:51
• Can you expect the output values from each iteration of the optimization? Is it ever non-zero? – Ryan J. Smith Feb 4 '15 at 7:02
• yeah, if I understand what you're saying. I'm starting with initial_theta = np.array([0,0,0]) and when calculating line by line I get a result of final_wealth = 0.1335, then I tried initial_theta = np.array([-0.1,0,0]) and got final_wealth = 0.0433. So that part is working properly (and same as my MATLAB solution). But I also tried changing the initial_theta to non-zero values and running the optimization function and ended up with that same non-zero initial_theta. So it seems that the optimization isn't running any iterations... – acmyers Feb 4 '15 at 7:26
• Sorry, that should say "inspect" rather than "expect". It seems like you caught the drift though. – Ryan J. Smith Feb 4 '15 at 7:28
• TheBlackCat solved it, but thanks for your help Ryan! – acmyers Feb 6 '15 at 7:15