I have a dataset of genes for which I'm trying to predict genes that cause a disease. Originally I was doing this with a multilabel classification. I had 3 groups: I labeled already known disease-causal genes as positive causal examples, I labeled genes with research in the disease as probable, and I labeled a final group as negative (with evidence they are non-causal).
I only have 50 known positive labeled examples, ~150 genes in the probable group, and ~250 in the negative group. I was struggling with the class imbalance so I have converted the problem into a regression with converting the labels to scores between 0 and 1.
However, my question is, I don't have exact guiding criteria to choose which label gets what score between 0 and 1 - can I test the model on several scoring intervals and take the top performing model? Or is this biased?
For example I've run a model where the 3 labels are converted to scores with equi-distant intervals (e.g. 0.9, 0.6 and 0.3, or 1, 0.5 and 0) they all perform almost the exact same. I tried scoring the genes in ways that I thought reflect the labeling criteria (from a biological perspective of how similar the criteria is for known and probable genes are). For example, scoring 1 for known/positive, 0.6 for likely and 0.1 for negative the models perform a bit better.
In general, the models do better about +0.02-0.05 better in nested cross-validation with performance metrics like r2 or MSE when the known and probable genes have closer scores together and the negative group is lower/further away in score - but I'm wondering if I can even use this, from a data science perspective is the converting some scores to be closer together and others to be further away biasing the model or is the +0.05 performance increase genuine? If I have a domain-based knowledge reason for why a known gene is 1 and a likely gene is 0.6 etc. is it then acceptable?