5
$\begingroup$

I'm working on multiclass logistic regression model with a large number of features (numFeatures > 100). Using a Maximum Likelihood Estimation based on the cost function and gradient, the fmincg algorithm solves the problem quickly. However, I'm also experimenting with a different cost function and do not have a gradient.

Is there a good way to speed up the calculation process? E.g., is there a different algorithm or fmincg setting that I can use?

$\endgroup$
2
  • 3
    $\begingroup$ What is this algorithm fmincg? Please, add some references to such algorithm in your question. $\endgroup$ – Rubens Jun 21 '14 at 6:51
  • $\begingroup$ What is your alternate cost function? The optimal optimization algorithm depends on it :) $\endgroup$ – nimish Jun 22 '14 at 12:03
3
$\begingroup$

If you do not have a gradient available, but the problem is convex, you can use the Nelder-Mead simplex method. It is available in most optimization packages, for example in scipy.optimize.

$\endgroup$
2
$\begingroup$

Conjugate Gradient -- the cg in Function MINimization (nonlinear) Conjugate Gradiant -- requires you to have a gradient function (or approximation) since that is a critical part of the algorithm itself: it needs to find the steepest descent direction quickly.

fminsearch implements Nelder-Mead, a nonlinear gradient-free method. Its convergence properties are not anywhere near as good.

What is your cost function? Are there approximations that are differentiable (pref. twice so you can use the very powerful quasi-Newton methods)?

$\endgroup$
2
$\begingroup$

I have been able to optimize very strange functions with simulated annealing, and it does not require a gradient. Instead, it uses random numbers in a way very similar to Markov Chain Monte Carlo, which helps it avoid getting stuck in local optima. A decent explanation that gives the intuition behind it can be found in this lecture: Simulated Annealing. scipy 0.14 includes this algorithm in its optimization module: scipy.optimize.anneal.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.