37
votes
Accepted
How to adjust the hyperparameters of MLP classifier to get more perfect performance
If you are using SKlearn, you can use their hyper-parameter optimization tools.
For example, you can use:
GridSearchCV
RandomizedSearchCV
If you use GridSearchCV,...
- 3,448
27
votes
How to set the number of neurons and layers in neural networks
The consideration of the number of neurons for each layer and number of layers in fully connected networks depends on the feature space of the problem. For illustrating what happens in the two ...
- 13.7k
16
votes
How to set the number of neurons and layers in neural networks
Very good question, as there doesn't exist an exact answer to this question yet. This is an active field of research.
Ultimately, the architecture of your network is related to the dimensionality of ...
- 5,326
12
votes
How to set the number of neurons and layers in neural networks
Short Answer: It is very related to the dimensions of your data and the type of the application.
Choosing the right number of layers can only be achievable with practice. There is no general answer ...
- 1,205
9
votes
Accepted
Which parameters are hyper parameters in a linear regression?
I like the way Wikipedia generally defines it:
In machine learning, a hyperparameter is a parameter whose value is
set before the learning process begins. By contrast, the values of
other ...
- 4,108
9
votes
Accepted
Which is first ? Tuning the parameters or selecting the model
You can tune parameters only if you have already trained the model, otherwise there is nothing to tune.
However, i've also read that model selection shoud be done before tuning the parameters.
...
- 538
7
votes
Accepted
How to choose the random seed?
TL:DR, I would suggest not to optimise over the random seed. A better investment of the time would be to improve other parts of your model, such as the pipeline, the underlying algorithms, the loss ...
- 14.4k
7
votes
Accepted
What makes a Tree-Structured Parzen Estimator "tree-structured?"
It means that your hyperparameter space is tree-like: the value chosen for one hyperparameter determines what hyperparameter will be chosen next and what values are available for it.
From a HyperOpt ...
- 10.8k
7
votes
What is the most efficient method for hyperparameter optimization in scikit-learn?
Optimization isn't my field, but as far as I know, efficient and effective hyper-parameter optimization these days heavily revolves around building a surrogate model. As models increase in complexity, ...
- 371
7
votes
Accepted
ROC AUC score is much less than average cross validation score
Your test score is incorrect. The ROC curve needs the probability scores from the model, not the class decisions. So replace
...
- 10.8k
6
votes
Which parameters are hyper parameters in a linear regression?
Hyper-parameters by definition are input parameters which are necessarily required by an algorithm to learn from data.
For standard linear regression i.e OLS, there is none. The number/ choice of ...
- 820
5
votes
How to set the number of neurons and layers in neural networks
Working with neural networks since two years ago, this is a problem I always have each time I wan't to model a new system. The best approach I've found is the following:
Look for similar problems ...
- 760
5
votes
Accepted
how to pass parameters over sklearn pipeline's stages?
I figured out how to do that by monkey patching ParameterGrid.__iter__ and GridSearchCV._run_search methods.
ParameterGrid.__iter__ iterates over all possible combinations of hyerparameters (dict of ...
- 231
5
votes
Accepted
GridSearchCV vs RandomSearchCV and How it works?
Imagine the following scenario:
...
5
votes
Accepted
Is a test set necessary after cross validation on training set?
In "The Elements of Statistical Learning" by Hastie et al the authors describe two tasks regarding model performance measurement:
Model selection: estimating the performance of different ...
- 5,260
5
votes
How are parameters selected in cross-validation?
Usually, the aim of K-fold cross-validation is to check how a model performs (both on average and how much it varies across folds) given some hyper-parameters setting. We then pick the "best"...
- 597
4
votes
How to adjust the hyperparameters of MLP classifier to get more perfect performance
As a complement to the very practical answer of @BrunoGL, I'd like to give a more theoretical answer. I'd like to suggest everyone trying to adjust hyperparameters of a simple Neural Network to read ...
- 2,377
4
votes
Accepted
Validation accuracy is always close to training accuracy
The fact that the training accuracy and the validation accuracy are close it is nothing to be concerned about. As you mention, it means your model are generalizing well.
The thing to be worried about ...
- 1,618
4
votes
When to use BayesianSearchCV and how it works?
As mentioned in that Kaggle notebook, you can use it pretty much as just a drop-in replacement for other search methods (grid or random). Bayesian searches still are random searches over a predefined ...
- 10.8k
4
votes
Why my network needs so many epochs to learn?
My opinion:
You should try to increase the learning rate of your model (or even other parameters of your optimizer - e.g. momentum).
To answer your questions:
Why the network is still learning after ...
- 7,758
4
votes
Accepted
New parameters in final training
Unpopular opinion: Second quickest way to overfit (next to data-leakage) is hyper-parameter optimization.
Why? You are assuming you wont have covariate-shift, while in most of the cases you can bet ...
- 5,519
4
votes
Accepted
Opinions on an LSTM hyper-parameter tuning process I am using
First of all you might want to know there is a "new" Keras tuner, which includes BayesianOptimization, so building an LSTM with keras and optimizing its hyperparams is completely a plug-in task with ...
- 2,664
3
votes
Accepted
Hyperparameter Optimization for a Machine Learning Algorithm
If your error stays at about 20%, it sounds like your features are not really helping.
It is likely the case that your relationship in data/features is not simple, so you need to allow your SVM model ...
- 14.4k
3
votes
Hyperparameter Optimization for a Machine Learning Algorithm
You are right, what has to be changed is the objective. You are currently using accuracy as a measure of how good your classifier is. Accuracy is not a good measure when you have class imbalance. For ...
- 5,921
3
votes
Accepted
Optimising Kernel parameters using training data in GaussianProcessRegressor of Scikit-learn
Given scikit-learn's API, you create a separate instance for each optimizer and compare the results to see which optimizer makes better predictions. It would looks something like:
...
- 19.5k
3
votes
Hyper parameters and ValidationSet
Not completely true. In validation set, we find the best hyperparameters, but not with the same parameters of the model. That is, for every value of the hyperparameters we run the training process and ...
- 5,921
3
votes
Automated Hyperparameter tuning
There are a number of methods to automate the optimization of your hyper-parameters, such as GridSearch and RandomSearch which the article you linked discusses briefly.
The main reason to choose one ...
- 1,712
3
votes
Accepted
How to make it possible for a neural network to tune its own hyper parameters?
I am curious about what would happen to hyperparameters when they would be set by a neural network itself
In general this is not possible as many hyper-parameters are discrete, so they are not ...
- 28.1k
3
votes
Accepted
overfit a Random Forest
Decision Trees are definitely easier to overfit than Random Forests. The averaging effect (see bagging) is meant to combat overfitting.
Other than that I think the default parameters will overfit.
...
- 7,758
3
votes
What is the most efficient method for hyperparameter optimization in scikit-learn?
You can take a look at auto-sklearn. That's an automated machine learning toolkit which is a direct extension of scikit-learn.
- 526
Only top scored, non community-wiki answers of a minimum length are eligible