A few short comments:
- "small" is relative and depends on the complexity of the model to be learned
- I don't understand how exactly you train your model after you found the best hyperparameter, could you clarify?
- in any case, you always have to evaluate the model performance based on a separate test dataset
Here are the answers to the additional information/questions:
After finding the best hyperparameter for the model, I then define a new model using those parameters and train that model with repeated k-fold cross validation, is this correct?
There is more than just one workflow for hyperparameter optimization and training/testing. In your case, your first run of k-fold cross validation (CV) is only used for determining hyperparameter. There, you use the held-out dataset to evaluate the performance of the candidate models. Once, you knwo the performance of all candidate models, you can choose the best-performing model, the so-called "final model". A strategy is, e.g., to use the candidate model that during CV gives -- on average -- the best performance.
The next step is to train the final model, e.g., for the purpose of prediction. Again, you have a number of options for evaluating the "goodness" of the training but all of them involve that you kept a dataset, the testing dataset, which has not been used in the process before. However, you can use everything MINUS the testing dataset (i.e., the previous training and also the testing folds) as training dataset for training the final model. For (cross) validation of the training process you can, as always choose between the hold-out method (where you split the whole dataset into a single training and validation set) or, e.g., the k-fold CV.
Regarding the separate test dataset, the model gives different results depending on the train test split, i assume this is due to how small the test set is, how can i mitigate this? Should i train the model on multiple train test splits?
This touches upon another subject: the sample and the population. Ideally, the training, validation or testing datasets should be representative for the population (i.e., the entire possible dataset) but most of the time we only have a small subset available, the sample. If the sample (e.g., the testing dataset) is large enough then we can use it for statistical analysis and use it to guess something about the whole population. The nice thing about k-fold CV is that this gives you a way to see if the folds are statistically representative. In your case, the dataset seems to be rather small indeed. There are several options three of which are: 1. try to analyze the dataset and see if it has a strong imbalance -- this might then require some (re)sampling stragey; 2. choose the value of k carefully as it determines the size of the folds (you read about k=5 or 6 is a good value but in your case you might want to choose a lower number); 3. get more data. In any case, training the model on multiple runs of train-test splits should not be done as in this case you would be using testing data that already has been used for training.
