0
$\begingroup$

I want to cross-validate a model that plays the card game below (see image).

I trained the model on a dataset of 1000 games, with the goal to maximise the profit of each game. It works great on the training dataset, but I want to k-fold validate it, to ensure that it generalises well.

How can I apply k-fold validation in this case?. This model is not a regressor. It is a classifier, with a variable reward/penalty moreover. So I cannot calculate the MSE of validation folds, there is no MSE here. How can I apply k-fold validation in this case?

enter image description here

$\endgroup$
1
$\begingroup$

So I cannot calculate the MSE of validation folds, there is no MSE here.

The question that you firstly need to ask is "how do I evaluate my model".

Since the model is a classifier, you can use the accuracy (number of correct classifications over the overall attempts) or the F-score as measures of your model's performance. Based on the chosen measure, you will thereafter evaluate the results of kfold-validation.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks, but accuracy is not my goal function, profit is my goal function. Accuracy would be equivalent to profit if profits/losses were always of the same amount, but as you can see from the table this is not the case. You can be accurate on many "poor" bets and inaccurate on few "rich" bets and still lose money. How can the F-score help me with this? $\endgroup$ – elemolotiv Aug 22 '18 at 21:20
  • 3
    $\begingroup$ Why not evaluate the profit on the validation fold, if that's the number you care about? Then you can estimate the variance of the profit across folds. $\endgroup$ – Dave Kielpinski Aug 22 '18 at 22:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.