Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I'm working on multiclass logistic regression model with a large number of features (numFeatures > 100). Using a Maximum Likelihood Estimation based on the cost function and gradient, the fmincg algorithm solves the problem quickly. However, I'm also experimenting with a different cost function and do not have a gradient.
Is there a good way to speed up the calculation process? E.g., is there a different algorithm or fmincg setting that I can use?
If you do not have a gradient available, but the problem is convex, you can use the Nelder-Mead simplex method. It is available in most optimization packages, for example in scipy.optimize.
Conjugate Gradient -- the cg in Function MINimization (nonlinear) Conjugate Gradiant -- requires you to have a gradient function (or approximation) since that is a critical part of the algorithm itself: it needs to find the steepest descent direction quickly.
fminsearch implements Nelder-Mead, a nonlinear gradient-free method. Its convergence properties are not anywhere near as good.
What is your cost function? Are there approximations that are differentiable (pref. twice so you can use the very powerful quasi-Newton methods)?
I have been able to optimize very strange functions with simulated annealing, and it does not require a gradient. Instead, it uses random numbers in a way very similar to Markov Chain Monte Carlo, which helps it avoid getting stuck in local optima. A decent explanation that gives the intuition behind it can be found in this lecture: Simulated Annealing. scipy 0.14 includes this algorithm in its optimization module: scipy.optimize.anneal.
