0
$\begingroup$

I've created my own neural network in python. I am tracking the performance of each run. What are some things I can investigate when my performance results are inconsistent? When I compare my network results against my expect results (during cross-validation) I can get everything from 0.46 to 0.51 all the way up to 0.56 and various numbers in between.

Is this a sign of bad data? Not enough training epochs? Need for a different learning rate?

$\endgroup$
2
  • 2
    $\begingroup$ Can you give more detail? How many classes are you predicting between? How much data are you using for training/cross validation/testing? How big is the input layer? How many hidden layers? What optimization are you using? etc. $\endgroup$ Aug 9, 2016 at 18:07
  • $\begingroup$ @JohnYetter Good questions, thanks! I am using small data sets, less than 5000 records each for test and train. 15 inputs to 1 hidden layer to produce 3 classification outputs. $\endgroup$ Aug 9, 2016 at 18:10

2 Answers 2

2
$\begingroup$

Before briefing about the other things, I would like to suggest you something good. It is always better to set a seed number in machine learning and deep learning problems to get a reproducible result and by that I mean that every time when you try to run your model anywhere, it can produce the same best result which you got on your machine while experimenting. So, you should always set numpy.random.seed(*anynumber*).

Now how to compare your result to your expected result? There are a number of parameters that you can look upon during training your model. Some of these are:

  • Validation accuracy: It is the one of the most important parameter to look upon for evaluating your model's performance. This gives you a fair idea if your model is overfitting or not. More the validation accuracy, the better is the model.
  • Validation loss: If your model is really good, then validation loss must decrease, at least initially until your best epoch hasn't been obtained. Lower the validation loss, better the model.
  • Optimizers: There are a number of optimizers like 'rmsprop', 'sgd', etc. which can be used for different types of problems but not all give the same performance. You should grid-search for a number of optimizers for your model if you are not specific about which is best for your model.
  • Number of training epochs: Depending upon the size of dataset, no. of training epochs differ but if you are not giving enough training to your model, then your model performance will suffer.
  • Training data: If you think your model is fine, but you are not achieving expected results, then you can try to train your model on a large dataset. NN require large dataset for achieving awesome performance.
$\endgroup$
1
  • $\begingroup$ If you use numpy to set the random seed you will be able to fool yourself into thinking you have a consistent performance. Not recommended IMO. $\endgroup$
    – Roko Mijic
    Dec 21, 2019 at 18:34
0
$\begingroup$

Data: It depends on how distinguishable your classes are. Like cats vs dogs is quite easy, while disgust expression vs angry expression is quite hard.

Epochs: Over-fit your model and do early stopping. To handle over-fitting, increase regularization or decrease the complexity of model by decreasing the number of hidden neurons / layers. See what's best for you given the time and performance.

Learning rate: Start with the largest learning rate that allows the model to learn. From there, experiment by dividing that learning rate by 10 or 2, observe the results, see which learning rate allows your model to converge faster and leads to higher accuracy.

There are still a lot of things to consider to achieve the maximum performance such as learning rate decay, gradient descent optimization algorithms, weight initializations, etc.

More observations = more productivity. Be mindful of the time and speed in choosing the best hyperparameters.

To know more, I suggest reading this by Michael Nielsen - Neural Networks and Deep Learning

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.