I build my first neural network! What's next?

Question

For an assignment we are given a dataset and we need to build a neural network to make new predictions.

Personal milestone; I built my first neural network!

I followed the following steps on my own data set:

https://machinelearningmastery.com/tensorflow-tutorial-deep-learning-with-tf-keras/

The initial set-up of a neural framework is fairly straight forward when following the steps in the tutorial. The next step is to tune the hyper parameters and here is where I get stuck. Important to note that my data set contains 22.000 observations with 160 parameters. There parameters are continuous variables, but also dummy variables.

I learned about the number of layers, the size of a layer, the activation function within a layer and the batch size. There are so many possible combinations and that's where the problem arises. Preferably I do a grid search and let my machine run overnight, so that I can find out which combinations of hyper parameters works the best.

However, then I still need to have a basic idea about the ranges of the parameters and the choices of the activation functions.

What did I do myself?:

I searched for autoML libraries and found the following:

https://levelup.gitconnected.com/10-python-libraries-for-automated-machine-learning-that-you-should-think-to-use-in-2023-730a2694f221

I am considering using autosklearn.classification, but is it advised to do so?

My question is:

• I have 0 intuition about tuning the hyper parameters (this is my first encounter with ML), how do I choose the right hyperparameters?
• It is advised to use the following code?:

import autosklearn.classification cls = autosklearn.classification.AutoSklearnClassifier() cls.fit(X_train, y_train) predictions = cls.predict(X_test)

Any advice would be very much appreciated!

Answer 1

In addition to the @brewmaster321's answer, in my opinion the hyper-parameters you need to care about are:

• Learning rate: especially if you use SGD as optimizer, which does not adapt the LR unlike Adam. Usually you want to grid search over a log-space, i.e. [$10^{-5}$, $3\cdot 10^{-4}$, $10^{-3}$, ...].
• Activation function. The most common are ReLU and leaky-relu. There are also some activations that are suggested for a given application and even neural net layer (e.g. tanh for recurrent layers.)
• Weight decay: used to regularize your NN. As for the LR, you want to search in log-space.
• #layers and #units/filters. These mainly impact the complexity of your NN. Basically, more layers and units (also called neurons) increase the number of learnable parameters (or weights) making your model bigger, larger (i.e. requires more storage space and memory), and slower (higher latency in inference, and slower training.) Indeed, these could increase the capability of the model to fit more complex data so reaching better performance. But notice that more parameters increase the chance of overfitting, and so you need to regularize more.

Also inspecting the learning curves (apart from metrics) may provide you hints about which hyperparameter you need to tune.

Answer 2

Welcome to the forum @Tim, and congratulations on your first neural network. If you're already using Keras, you can use the Keras tuner on your network instead of using a separate framework. The Keras tuner documentation is here: https://keras.io/keras_tuner/https://keras.io/keras_tuner/ and there's a tutorial here: https://www.tensorflow.org/tutorials/keras/keras_tuner. You are correct that you need some rough idea of valid values for your hyperparameters to start with, but assuming you have a working network already, you can start by incrementing/decrementing your working values for your hyperparameters.