I have a dataset that I curated from a game that I play. There are currently 130 instances (i.e. players) and an innumerable number of features. Experience tells me <10 features would be sufficient. There are numerous outliers, and correlation among the features.

The two main goals are (1) gain insight on what makes the winners win and the losers lose, and (2) group players into various categories so I can develop a strategy suited to each one. As for point (1) I'm FAR more interested in why winners win since I wan't to emulate them and it is often obvious why losers lose. For now let's focus on the first goal of classifying players.

One big problem is that the classification labels (i.e. whether someone is a winner or loser) are very noisy. In fact, in my database there are some players labeled as losers who I know are winners and vice-versa. This is because an enormous sample is needed to accurately classify someone. The labels seem mostly correct.

Another big challenge is feature selection -- there are a lot to choose from.

Let's ignore the first issue for now, and for the second I chose 13 features I believe to be important.

I used a gradient boosting and RF classifier and optimized the hyperparameters to obtain the following learning curves (N.B. the dataset is balanced so accuracy seems like a fine metric, but might be worthwhile to also optimize the recall for winners since that is what I'm interested in?) enter image description here enter image description here

In any case my conclusion from this graph was that more data is needed, which is not surprising. This was the best model I could do with the dataset and labels I had. When I inspected the errors from this model I found, e.g., that a lot of the false positives (i.e. classified as winners but were losers in my database) aren't actually misclassifications. That is, these players I know are actually winners but in my database are losers because the sample was insufficient in determining their classification. So now let us address the noisy labels because it appears to be an issue.

It is possible to obatin more data, but it is costly (financially and time). With that in mind I wanted to make as much progress with the data I have since it isn't always possible to get more data.

So, now we return to the issue of the labels and their accurarcy. I could hand label ~20-30 instances confidently and then perform semi-supervised clustering. I don't have a lot of knowledge here so perhaps I will come back to this. Thinking on the problem I guess my central hypothesis is that it is possible to classify players as winners and losers based on a handful of features -- this is what my knowledge tells me. With that in mind, it shold be possible to cluster the dataset into two main clusters corresponding to winners and losers. Presumably each cluster could be broken down further (e.g. big winners, big losers, etc).

So, I perform k means clustering for a range of k, calculate the average silhouette score and plot the results. It appears that k = 2 is best, and inspecting the clusters it appears that they are split into winners and loser!! enter image description here

These labels are much more accurate. Now, using these labels I perform classification using a gradient boosting classifier and SVC to obtain the following learning curves. N.B. I didn't do any hyperparameter optimization yet, this is with the default parameters. enter image description here enter image description here

These curves look quite good, almost too good (?) I'm still learning to interpret these graphs, but I believe the SVC one looks better. In any case, for the final step of this, I look at the feature importances from the gradient boosting classifier and the three most important align with my intuition. In particular, the feature it says is most important is also the one I believed was most important beforehand, so that is very reassuring.

Questions/Concerns: (1) Is this workflow acceptable, i.e., clustering players to obtain labels and then using them for classification? I'm just worried that this is a bit tautological -- if you can find a way to cluster a dataset, then you should be able to easily classify it? (2) The reason for using a gradient boosting classifier was to gain insight on feature importance, however lately I have been reading that the notion of feature importance is unreliable and uninformative. In any case, assuming the model is correct I need to interpret why it is making the classifications it is making.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.