5
$\begingroup$

I'm training a NN with 8 features and 8000 training examples with a single output (0, 1) using the scipy.optimise CG algorithm and the results are somewhat inconsistent. The goal is to get the NN to be as 'precise' as possible (recall doesn't really matter too much) so I've set the threshold for y value quite high (0.75). Most of the time it gets a precision of around 80%, however sometimes it fails (using exactly the same parameters, lambda etc..) to generate any outputs which are above the 0.75 threshold, meaning the precision equals 0.

I've successfully trained NNs before without these unusual results (albeit the goal was a somewhat more conventional multi-class classifier with many more features).

I'm wondering if the training NNs with fewer features increases the chances of it getting stuck at a local optima; or getting stuck at local optima has a more significant impact on NNs with fewer features?

Any thoughts on what's going on!?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ It's a common problem, many techniques invented to try and address it - momentum, adaptive learning rates, batch solvers using second-order approximations etc. Fundamentally, gradient descent is a ropey choice for optimising a complex function, but the best option with neural networks is usually some form of it with some fixes to make it more bearable. $\endgroup$
    – Neil Slater
    Oct 28, 2014 at 16:57
  • $\begingroup$ Hi Neil, thanks for the comment. Can you suggest other areas I could explore other than gradient descent algorithms? $\endgroup$
    – denton101
    Nov 1, 2014 at 7:36
  • 1
    $\begingroup$ I don't really think there are any serious contenders for nn optimisation on large complex networks. For small networks, a genetic algorithm search can work (see en.wikipedia.org/wiki/Neuroevolution_of_augmenting_topologies or examples) $\endgroup$
    – Neil Slater
    Nov 1, 2014 at 8:41
  • 1
    $\begingroup$ NN initialization is also an open issue. $\endgroup$
    – Chen Guo
    Nov 28, 2014 at 6:07

1 Answer 1

Reset to default
5
$\begingroup$

Ok all sorted - Bit embarrassing but forgot to normalise the data!

Improve this answer
$\endgroup$
1
  • $\begingroup$ Wow thanks for the pointer. I made the same mistake! $\endgroup$
    – Dylan Kerler
    Mar 2, 2021 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.