Tag Info

5

In "The Elements of Statistical Learning" by Hastie et al the authors describe two tasks regarding model performance measurement: Model selection: estimating the performance of different models in order to choose the best one. Model assessment: having chosen a final model, estimating its prediction error (generalization error) on new data. ...

3

As I was playing with this problem, Djib just wrote an answer which is certainly better than whatever I could have come up with. To illustrate Djib's point, here is a small demonstration that as soon as there are more than 2 classes there's no value of $k$ which guarantees the absence of tie (except if $k=1$ of course). By definition we have $k=|c_1| + |c_2| ... 3 Your idea isn't wrong, however in k-NN there always might be a case where you have the same number of votes for 2 or more classes (e.g. you have$k=6$and you have 3 samples of one class vs 3 of another). With your solution you are just overcoming one very small case of ties and the$k$that you choose might not be the optimal$k$for classification, which ... 3 The$k$-fold cross-validation (CV) process (method 2) actually does the same thing as method 1, but it repeats the steps on the training and validation sets$k$times. So with CV the performance is averaged across the$k$runs before selecting the best hyper-parameter values. This makes the performance and value selection more reliable in general, since ... 2 This idea of using a small sample to the data set to search for the hyperparameters is called multi-fidelity methods. A good starting point is the book by Frank Hutter, Lars Kotthoff, Joaquin Vanschoren Automated machine learning: methods, systems, challenges which is open access. 2 So, the question talks about how to treat transformation choices as hyper parameters. How I would go about it is the following: Use one baseline model architecture for the data and then repeat the following: Instantiate the baseline model (effectively make sure all of the weights are initialised) Create the transformed dataset Train the model Compute ... 2 Note that RMSE is an easy to understand metric. Its the Root of the Mean Squared Error. So this is just how is the typical error. If your target is something like how big is a building, and the mean of the target its 100m, then having an error of 0.3m its nothing. On the other hand if you predict the size of insect, and your target mean is around 0.1m then ... 2 There's quite a lot of features for the number of instances, so it's indeed likely that there's some overfitting happening. I'd suggest these options: Forcing the decision trees to be less complex by setting the max_depth parameter to a low value, maybe around 3 or 4. Run the experiment with a range of values (e.g. from 3 to 10) and observe the changes in ... 2 I assume its crashing because not enough RAM. So I also assume your data is quite big. Your search grid is quite big. So this will definitely take some time. In order to speed up the training and overloading the RAM. You can fit the model in a subsample of data. Theoretically if your data is big enough and you sample it, when you use the whole model the ... 2 1. what are the rows before the first red row? I thought it may be the combinations of parameters but that doesn't make a lot of sense because those are not enough Parameters, which are the candidates of the CV are printed 2. What is the meaning of the row between the red rows? Why those parameters are there? is this after one CV? It is printed after the ... 2 As far as I know you cannot add the model's threshold as hyperparameter but in order to find the optimal threshold you can do as follows: make a the standard GridSearchCV but use the roc_auc as metric as per step 2 model = DecisionTreeClassifier() params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],&... 2 If you're seeing performance that is much better on the validation than the unseen test data, then that is suggestive of some sort of overfitting or, if not, that the data do not come from the same distribution. That could mean that your test images are very different from the training and validation data, for example. First, I'd double check the data to ... 2 Essentially, the function of your testset is to evaluate the performance of your model on new data. It mimics the situation of your model being put into production. The validation set is used for optimizing your algorithm. Personally I would recommend tuning your algorithm using your validation set and using the hyperparameters of the training epoch with the ... 1 So do we still need to learn how to do hyperparameter tuning If you're saying this based on the context of acquiring a new skill, then go for it. It's always a good thing to get an idea an idea of how hyper-parameter testing is done for real. In addition to sagemaker you can use tools like weights and biases 1 In general there's no way to know the best values to try for a parameter. The only thing one can do is to try many possible values, but: this mathematically requires more computing time (see this question about how GridSearchCV works) there is a risk of overfitting the parameters, i.e. selecting a value which is optimal by chance on the validation set. 1 This is very nearly a duplicate of Is a test set necessary after cross validation on training set?, but I think it's worth addressing specifically this part of your question: the hyperparameter searching is like every time training a model from scratch using a combination of hyperparameters and pick the best combination, and it's not like that the model is ... 1 You simply need to see the dynamic of change in the incoming data. The need for retraining is a direct function of change in data distribution. If new data, theoretically, does not change the distribution then there is no need, however in practice that is simply impossible. So how much change in the distribution of data (both input and output) is a threshold ... 1 Hello and welcome to the site! You can use list(range(1,100)) to get what you want. However, questions like this (how to achieve something in Python) are more suitable for stackoverflow. The community here focuses on data science related questions, as the name suggests. 1 Yes, you can make an AI for that. By AI, you mean algorithm which finds hyperparameters efficiently. There are many ways to find hyperparameters which would then come under the AI like using bandits to find hyperparameters, Bayesian methods to find hyperparameters and many other methods exist. Search for hyperparameter optimization and you will find a ton of ... 1 In a pure random search, 60 points is often given as a rule of thumb, because provably with probability 95% such a search finds a hyperparameter combination in the top 5%. However, that 5% is as a percent of the volume of the search space, so giving much-too-broad a search space, the best 5% might not be a fantastic score for the model. So it does seem to ... 1 To add a couple of references, to the already good answers, on the problem of selecting the optimum$k$for$k-NN\$. How to find the optimal value of K in KNN? Then how to select the optimal K value? There are no pre-defined statistical methods to find the most favorable value of K. Initialize a random K value and start computing. Choosing a small value of ...

1

Have a look at this blog post: https://towardsdatascience.com/hyperparameter-tuning-the-random-forest-in-python-using-scikit-learn-28d2aa77dd74 Ideally you should optimise the hyperparameters jointly and not one after the other. More importantly, you should be doing cross validation. Consider also the RandomizedSearchCV described in the post.

1

Since the various hyperparameter are related you cannot be sure that - for instance - a tuned value for min_samples_split from a model where all other hyperparameter are set to default values, will be generally optimal. When you have a situation where you limit the depth of trees (by max_depth assuming you use sklearn), min_samples_split may be a non-binding ...

1

In general, the max depth parameter should be kept at a low value in order to avoid overfitting: if the tree is deep it means that the model creates more rules at a more detailed level using fewer instances. Very often some of these rules are due to chance, i.e. they don't correspond to a real pattern in the data. Overfitting is visible in your graphs from ...

1

I was wondering if i'm doing a GridsearchCV on 10-fold, getting the best parameters, and then using those parameters evaluating the performance on 10-fold - is that "legal" or overfitting? am i suppose to run the best parameters on the entire data ? or can i use 10-fold again? I'm pretty sure you won't go to prison for it ;) But it would be ...

1

If you have the time on hand, you could simply measure the time taken for all combinations of hyper parameter values in a Grid Search, preferably with repetition. It's unlikely that any theoretical analytical expression will provide adequate accuracy for predicting the compute cost, as there as so many factors that contribute noise to the compute time. You ...

1

A tip: Dont A trick:Dont The reason? Machine learning scientific methodology is based on cross-validation. Almost all papers (and i put the almost because of yes) select everything based on cross-validation and not in previous knowledge. Xgboost is particularly more complicated because it has a lot of math involved. For a simpler case, lets say that you have ...

1

You can access the GPU by going to the settings: Runtime> Change runtime type and select GPU as Hardware accelerator.

1

What shepan6 is suggesting is basically to manually search for the best "transformation choice hyperparameters" by trying them all and seeing what performs best. This is a good idea (I upvoted), but if you want to go further, you can use a package like hyperopt and manually define an "objective" function that accepts a parameter that ...

1

So that we are on the same page, some prerequisites Suppose we had only 2 splits train and test. Now when we will tune our hyperparameters using the test split, we are trying to increase the accuracy(or any other metric). Though our model is not trained on the test set, but we are making it perform well on the test set, in a way the model gets the ...

Only top voted, non community-wiki answers of a minimum length are eligible