Machine Learning on Small Dataset with Huge Variation

Question

In 2018 the NCAA allowed players to transfer school more freely, so now teams want an advantage by getting transfer players. I want to create a model that can predict which transfer players can be good if they transfer to my favorite team. Unfortunately, due to various constraints like Covid kicking out the 2020 season, and NCAA rules, the number of samples where a player has had play time for an old team and then transferred to a Power 5 school, is close to around 40 players. However, these 40 players have had different outcomes where some succeed and some don't. Additionally, these 40 players can be subsetted to have around 9 different play styles. Ie if there is a feature like points some players don't score much but they play because they have good defense and that shows in other features like Steals; but players who score a lot usually tend to not get a lot of steals. However, both players can be successful despite different variations in data.

So how would you guys approach this solution?

I thought of doing PCA on the data because I have like 80+ cols and then doing a sci-kit learn neural network on the PCAs, what are your thoughts on this solution does it address issues with my data?

Answer 1

Let me start of by saying that the project sounds really cool.
But here's the but: it is tremendously ambitious to accomplish what you want with the data you mention.
Here's some stuff I think you would really need to consider, I'm sorry for the long post, but maybe it gets your creative juices flowing!

What exactly do you want to know?
You say you want to predict good transfers for your favorite team. This means you'll need to define what you mean by 'good for the team'.

• Do you want to fill a gap left by a player that is leaving?
• Do you want to fire and replace the current weakest link?
• Do you want to replace an expensive player with a cheaper young transfer with similar performance stats?

Having a clear research question with some way to measure it will provide good directions into what data you need. Then, if you don't have the data, you can look for another question, or wait/look for more data :)

What information is in your data?
Predicting which transfer players would be good for your favorite team requires that you consider a lot of potential transfers. A model can only ever predict transfers out of the sample you give to it. Therefore, you would need data on all possible transfer players - or at least the majority of them - in order to say anything meaningful.
I know little to nothing about football, but let's say the population of potential transfers is 10 x your sample size. This could result in the best possible predicted transfer player still being worse than 90% of available alternatives in the real world.

My advice: do a thorough exploratory analysis
I would recommend you to do an exploratory analysis on this 'early sample' of college transfer players. Do you see differences in their performance statistics or the teams after the transfer? And can you be sure that these are not due to random circumstances (e.g., the other team's star player being quarantined - the typical 'broken leg' case).
Ideas could be:

• some nice insightful visualizations about general trends within this sample of ~40
• an in-depth case study for your favorite team

It is always especially interesting to look for things that surprise you, because then it is very likely that others will be suprised by it as well. And these things are not only very fun to discover, they are also quite publishable on a blog or website.

Hope this helps!