I have one dataset with $1500$ data points, each with $\sim 23 000$ features (gene expression data, if that matters). However, I've split this dataset into a labelled training set of size 1000, and a test set for which I don't have the labels of size 500.
My goal is to train a model on the first set in order to obtain the best possible balanced accuracy when predicting the labels on the test set.
By simply training a model on the training set, I'm not able to reach a very high score (when performing stratified $k$-fold cross-validation on the training set, as I cannot test my performance on the test set), regardless of which type of classifier I've used various models: SVM, decision tree, random forest, NN, LDA, QDA, KNN, AdaBoost, XGBoost. So far, the best performing are an MLP model and a (boosted) decision tree. However, I suspect that if I somehow find a way to give more weight to data in the training set that lies close to the data in the test set, I should be able to make a more performant model. However, I'm unsure as to how I should do this.
- I cannot get my hands on larger amounts of data.
- I cannot see my true score on the test set, since I have no access to it.
- I can use any machine learning model (in Python or R), though I prefer the former by far (using
- Due to time constraints, I can't really test all possible models (+ all various data transformations, + all possible parameters).
What I need help with
There are various things I could use help with:
- How can I make data points which are close to the ones in the test set be worth more (theoretically, and in practice in my code)?
- Is my cross-validation performance evaluation technique sound? If not, how should I adapt it? This is doubly important, because as part of the competition I have to evaluate the balanced accuracy I will obtain on the test set, and this prediction is taken into account when computing my final score (and needs to be as close to its real value as possible). I suppose LOO cross-validation might perform well here, but would probably take far too long to train.
- If any, what classifiers tend to perform well with the type of data I have? In general, how should I go about finding the optimal models/parameters/data transformations for a given problem?