If you are using SKlearn, you can use their hyper-parameter optimization tools.
For example, you can use:
If you use GridSearchCV, you can do the following:
1) Choose your classifier
from sklearn.neural_network import MLPClassifier
mlp = MLPClassifier(max_iter=100)
2) Define a hyper-parameter space to search. (All the ...
The consideration of the number of neurons for each layer and number of layers in fully connected networks depends on the feature space of the problem. For illustrating what happens in the two dimensional cases in order to depict, I use 2-d space. I have used images from the works of a scientist. For understanding other nets like CNN I recommend you taking a ...
Very good question, as there doesn't exist an exact answer to this question yet. This is an active field of research.
Ultimately, the architecture of your network is related to the dimensionality of your data. Since neural networks are universal approximators, as long as your network is big enough, it has the ability to fit your data.
The only way to truly ...
Short Answer: It is very related to the dimensions of your data and the type of the application.
Choosing the right number of layers can only be achievable with practice. There is no general answer to this question yet. By choosing a network architecture, you constrain your space of possibilities (hypothesis space) to a specific series of tensor operations, ...
You can tune parameters only if you have already trained the model, otherwise there is nothing to tune.
However, i've also read that model selection shoud be done before tuning the parameters.
Before tuning you need to do some kind of pre-processing before tuning the parameters.
Usually your pipeline will consist of:
Get Data and Clean It.
Do some EDA ( ...
Your test score is incorrect. The ROC curve needs the probability scores from the model, not the class decisions. So replace
y_predicted = grid_clf.predict(X_test)
y_predicted = grid_clf.predict_proba(X_test)[:,1]
TL:DR, I would suggest not to optimise over the random seed. A better investment of the time would be to improve other parts of your model, such as the pipeline, the underlying algorithms, the loss function... heck, even optimise the runtime performance! :-)
This is an interesting question, even though (in my opinion) should not be a parameter to optimise.
Optimization isn't my field, but as far as I know, efficient and effective hyper-parameter optimization these days heavily revolves around building a surrogate model. As models increase in complexity, they become a more opaque black box. This is the case for deep neural nets and presumably complex trees as well. A surrogate model attempts to regress the ...
I like the way Wikipedia generally defines it:
In machine learning, a hyperparameter is a parameter whose value is
set before the learning process begins. By contrast, the values of
other parameters are derived via training.
On top of what Wikipedia says I would add:
Hyperparameter is a parameter that concerns the numerical optimization problem at ...
Working with neural networks since two years ago, this is a problem I always have each time I wan't to model a new system. The best approach I've found is the following:
Look for similar problems that have also been modeled with feed-forward networks and study their architectures.
Begin with that configuration, train the data set and evaluate the test set.
Hyper-parameters by definition are input parameters which are necessarily required by an algorithm to learn from data.
For standard linear regression i.e OLS, there is none. The number/ choice of features is not a hyperparameter, but can be viewed as a post processing or iterative tuning process.
On the other hand, Lasso takes care of number/choice of ...
It means that your hyperparameter space is tree-like: the value chosen for one hyperparameter determines what hyperparameter will be chosen next and what values are available for it.
From a HyperOpt example, in which the model type is chosen first, and depending on that different hyperparameters are available:
space = hp.choice('classifier_type', [
The fact that the training accuracy and the validation accuracy are close it is nothing to be concerned about. As you mention, it means your model are generalizing well.
The thing to be worried about is the low training accuracy. It seems that your model is under-fitting the data but with low variance. This is the typical scenario of a model with high bias ...
You should try to increase the learning rate of your model (or even other parameters of your optimizer - e.g. momentum).
To answer your questions:
Why the network is still learning after so many epochs (and so slowly)? It is a reasonable behaviour? Do I need to run the model for 2000, or even 3000 epochs to get the best macro f1 score? It risks ...
If your error stays at about 20%, it sounds like your features are not really helping.
It is likely the case that your relationship in data/features is not simple, so you need to allow your SVM model more flexibility and/or train for longer. This will of course bring in the danger of overfitting, but should improve things... you'll need to try out a few ...
You are right, what has to be changed is the objective. You are currently using accuracy as a measure of how good your classifier is. Accuracy is not a good measure when you have class imbalance. For this reason, other objectives have to be used. AUC is a better measure, and so is the log-loss, although not that interpretable. However, both AUC and log-loss ...
I am curious about what would happen to hyperparameters when they would be set by a neural network itself
In general this is not possible as many hyper-parameters are discrete, so they are not differentiable with respect to any objective. For example, this applies to layer sizes, number of layers, choices of transfer functions. This prevents using any form ...
Not completely true. In validation set, we find the best hyperparameters, but not with the same parameters of the model. That is, for every value of the hyperparameters we run the training process and obtain the loss for that set of hyperparameters, and we select the hyperparameters values with the lowest validation loss.
Ridge regression example: ...
Given scikit-learn's API, you create a separate instance for each optimizer and compare the results to see which optimizer makes better predictions. It would looks something like:
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF
gp1 = GaussianProcessRegressor(kernel=RBF, optimizer=optimizer1)
There are a number of methods to automate the optimisation of your hyper-parameters, such as GridSearch and RandomSearch which the article you linked discusses briefly.
The main reason to choose one over the other is if you want the best possible parameters, and don't care how long it takes to get them: go for GridSearch. On the other hand, if you don't ...
I figured out how to do that by monkey patching ParameterGrid.__iter__ and GridSearchCV._run_search methods.
ParameterGrid.__iter__ iterates over all possible combinations of hyerparameters (dict of param_name: value). so i modified what it yields (one configuration of hyperparameters params) by adding "km__nbr_features" equal to 'tfidf__max_features':
Just to add to others here. I guess you simply need to include a early stopping callback in your fit().
from keras.callbacks import EarlyStopping
# Define early stopping
early_stopping = EarlyStopping(monitor='val_loss', patience=epochs_to_wait_for_improve)
# Add ES into fit
history = model.fit(..., callbacks=[early_stopping])
Unpopular opinion: Second quickest way to overfit (next to data-leakage) is hyper-parameter optimization.
Why? You are assuming you wont have covariate-shift, while in most of the cases you can bet on it. Hence optimising too much on train (available data) will be ruin.
The most reasonable assumption (that we have to make sure it stands) is that 60% of the ...
I had the same problem. It seems to be a bug in keras that occurs with nested arrays as parameters for the grid search. I was able to solve it by nesting tuples instead of arrays.
So try to change
layers = [,[50, 20], [50, 30, 15], [70,45,15,5]]
layers = [(50,),(50, 20), (50, 30, 15), (70,45,15,5)]
(add a ,(comma) when only 1 value is available in ...
The $k$-fold cross-validation (CV) process (method 2) actually does the same thing as method 1, but it repeats the steps on the training and validation sets $k$ times. So with CV the performance is averaged across the $k$ runs before selecting the best hyper-parameter values. This makes the performance and value selection more reliable in general, since ...
In "The Elements of Statistical Learning" by Hastie et al the authors describe two tasks regarding model performance measurement:
Model selection: estimating the performance of different models in order to choose the best one.
Model assessment: having chosen a final model, estimating its prediction error (generalization error) on new data.
Hyperparameter optimization follows the same rules as model selection. Each set of hyperparameters effectively represents a different model you are considering, so the data you use to fit the model with some set of hyperparameters needs to be different from the data you use to evaluate which set of hyperparameters you want to ultimately use. A common ...
Scikit-learn package has a limited selection of optimizers. The scipy package has many more optimitizers, including trust-region-reflective algorithm. You would have to use another third party package for Firefly algorithm.
Count of epochs is also a hyper-parameter. However, if you meant to ask what to choose to work upon, whether increasing epochs or some other methods like feature engineering , then below is my answer.
Increasing number of epochs is often attributed as hammer stroke to train the model where you yourself often don't have to think much about the data and ...