New answers tagged cross-validation
0
If you use the scikit-learn GridSearchCV class (from sklearn.model_selection) together with the scikit-learn wrapper in keras, you can get your final model refit on the whole training set directly via the best_estimator_ attribute (i.e. the model instanced with the best hyperparms found in the CV process) already refit with the whole training dataset.
I have ...
1
As I understand them, Bayesian optimization approaches are already somewhat robust to this problem. The evaluated performance function is usually(?) considered noisy, so that the search would want to check nearby the "best solution" $h$ to improve certainty; if it then finds lots of poorly performing models, its surrogate function should start to ...
1
One option is not to measure the performance of the hyperparameters on the loss function of the training data but measure performance of the hyperparameters on the elevation metric on the validation data. The end goal of the most machine learning systems is the ability to predict on unseen data. Focusing on "best solution" as measured by loss ...
0
In short, k-fold CV is not about over-fitting. Your samples can never be ideally identical, so you can only conclude that your error is mean±std.
If your model training process is iterative, then you can detect overfitting by checking test score over the course of training.
If you're making hyper-parameter search with k-fold CV, perhaps with many steps, then ...
1
Here are some ideas of things to try:
I would try to investigate the problem by not using the 'refit' option. Run the Grid Search CV yourself, obtain the best parameters and train a new model on all of the training data after you input those best parameters. This is just to make sure there is nothing funky going on with SKlearn.
Try using another metric ...
0
How are you saving your best weights for your models? If you have a checkpoint that is evaluating the loss/accuracy of your validation set instead of your training set, then you will end up with weights that overfit to the validation set and could do poorly against the test set. Not sure what kind of setup you have though so can you tell me how you save the ...
-1
I just did this:
union_data = td1.unionAll(td2).unionAll(td3).unionAll(td4)
worked like a charm
3
The way most people gain an initial understanding of label smoothing (and what most common explanations have to say on the subject) plays a great role in how one would approach this question.
At first glance, label smoothing is exactly what the name suggests: we modify the labels or some portion of them in order to get a better, more general, more robust ...
0
Label Smoothing is a regularizer technique that is applied to target value so that the model can learn the data well without overfitting. There is no need to do label smoothing for validation.But even if you do it, it won't be problem.
0
AFAIK label smoothing comes into picture while calculating the loss while training. There is no loss computation during validation.
0
It looks like there are very few loan defaults in your dataset, so xgboost is learning to predict 0 for all inputs regardless of the hyperparameters you choose. Try sampling from your data so that the classes are more balanced (e.g. >10% of your data points are defaults) and see what happens.
Other options for dealing with highly imbalanced classes can be ...
0
Pass your clf object to the function provided in the website. It should work.
0
You can understand whether your model is overfitting or underfitting by the difference in the graph of your train and validation score. If your train score(performance not loss) is low and so is val score then your model is underfitting. On the other hand if your model is overfiiting you will have high training accuracy but your validation score will be low ...
1
The error is caused by passing a numpy array into a function that expects an integer value. read_csv() will read a file, and create a numpy array from the data inside. You can slice off the column of the numpy array that you want to use, convert it to a list and then pass this one by one into classes[]
# this should work, but change the value of 12 to the ...
0
I'm not sure if there's a question here, but I'll add some comments.
Firstly, if you can get it in the wild, always work with balanced data. However, if you are going to manually create a "balanced" data set yourself, make sure that the selection criteria that you use to create that data is appropriate. As an example, choosing the 100k most recent ...
Top 50 recent answers are included
