I'm trying to use a particular cost function (based on doubling rate of wealth) for a classification problem, and the solution works well in MATLAB. See https://github.com/acmyers/compareCostFXs

When I try to do this in Python 2.7.6 I don't get any errors, but it only returns zeros for the theta values.

Here is the cost function and optimization method I've used in Python:

def costFunctionDRW(theta, X, y):

    # Initialize useful values
    m = len(y)
    # Marginal probability of acceptance
    marg_pA = sum(y)/m
    # Marginal probability of rejection
    marg_pR = 1 - marg_pA

    # =============================================================
    pred = sigmoid(np.dot(X,theta))
    final_wealth_individual = (pred/marg_pA)*y + ((1-pred)/marg_pR)*(1-y)
    final_wealth = np.prod(final_wealth_individual)
    final_wealth = -final_wealth

    return final_wealth

result = scipy.optimize.fmin(costFunctionDRW, x0=initial_theta, \
                   args=(X_array, y_array), maxiter=1000, disp=False, full_output=True )

Any advice would be much appreciated!

  • 2
    $\begingroup$ What type of data is y? Is it an int or a float? Do you use from __future__ import division? $\endgroup$ – TheBlackCat Feb 5 '15 at 9:36
  • $\begingroup$ y is an int... changed it to a float and solved the issue! thanks a ton! $\endgroup$ – acmyers Feb 6 '15 at 7:14
  • $\begingroup$ Great! I have added the answer as an answer below, please mark it as correct so others can find it more easily. $\endgroup$ – TheBlackCat Feb 6 '15 at 9:42

The problem is with division on python 2.x. In python 2.x, division involving two integers produces an integer result. So 1/2==0. Python 3.x does not have this problem, 1/2==.5.

There are two ways to avoid this. First, you can always convert one value you a float. So 1./2==0.5 and 1/2.==0.5. However, you have to remember to do this everywhere, and if you forget it can lead to hard-to-find errors.

The more reliable method is to always put this at the top of your code: from __future__ import division. This will switch python 2.x to the python 3 behavior, so 1/2==.5. In python 3.x it does nothing, so it also makes your code python 3.x compatible in this regard.

| improve this answer | |

If your number of cases is large, you may be running into a problem of numerical underflow in the line

final_wealth = np.prod(final_wealth_individual)

If each value of final_wealth_individual is between 0 and 1, multiplying them all together can lead to a result that is too small to represent as a floating point number, resulting in a value of 0.

To address this issue, take the log of final_wealth_individual and add them together instead of multiplying. Note that this will cause final_wealth to be negative, so you will not need to multiply it by -1 as you are currently.

| improve this answer | |
  • $\begingroup$ No luck with this solution. I'm also only using 100 examples/cases. $\endgroup$ – acmyers Feb 4 '15 at 6:51
  • $\begingroup$ Can you expect the output values from each iteration of the optimization? Is it ever non-zero? $\endgroup$ – Ryan J. Smith Feb 4 '15 at 7:02
  • $\begingroup$ yeah, if I understand what you're saying. I'm starting with initial_theta = np.array([0,0,0]) and when calculating line by line I get a result of final_wealth = 0.1335, then I tried initial_theta = np.array([-0.1,0,0]) and got final_wealth = 0.0433. So that part is working properly (and same as my MATLAB solution). But I also tried changing the initial_theta to non-zero values and running the optimization function and ended up with that same non-zero initial_theta. So it seems that the optimization isn't running any iterations... $\endgroup$ – acmyers Feb 4 '15 at 7:26
  • $\begingroup$ Sorry, that should say "inspect" rather than "expect". It seems like you caught the drift though. $\endgroup$ – Ryan J. Smith Feb 4 '15 at 7:28
  • 1
    $\begingroup$ TheBlackCat solved it, but thanks for your help Ryan! $\endgroup$ – acmyers Feb 6 '15 at 7:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.