How to handle ID variables when splitting data for machine learning?

Question

I'm new to machine learning, and I'm working with some international head-to-head sports competition data. I used relational data creation techniques in tidyverse to join several data sources to create an event based dataset where each row is the outcome of a unique match up between 2 teams and their measurable traits, with the obvious goal of finding the importance of those traits on the outcome.

Note = In general, I'm trying a few different ways to organize the data to get hands on experience with creating and analyzing data sets effectively, so if how I set it up isn't how you'd do it, don't go too hard on me. Specifically with have a repeat in variables based on home versus away, so that I could try to get all the event data into one and only one observation (example = "average_speed_home" and "average_speed_away"). I know and will try other data configurations, but any suggestions related to that would be nice, but it's not the main reason I'm asking for help.

My main question is related to what I should do with ID variables that I used to create the data set in relation to data splitting for machine learning. I've read on a few posts that I should keep those variables for the data splitting because it could create bias if I don't. But, the ID variables I have, aren't really factors that I want to included when creating my models.

Specifically, I used Home and Away variables for certain teams in the matchup, so I could pivot wider and include all the data for each event in one observation, but those team differences are already shown in the other variables that are dedicated to to either home or away for a certain trait, such as "average_speed_home" and "average_speed_away". The Home and Away variables now just say a Nationality. Since several of the yearly competitions are included in the data set, I wouldn't want to analyze the nationality impact on the results because the composition of the teams change quite frequently and the results might be biased due to recency of success (and also all the actual skills and performance metrics are included as their own variables).

Also, I'm unsure about whether to leave the match # ID variable as well because each line is a unique value that is essentially just an observation count.

Does the distribution of the data matter for data splitting when the data is event-based and standardized such that the only differences would be what actual National Teams faced each other in the matchup? And if I'm only looking for the numerical values of certain skills and their impact on the outcome, should I worry about including the ID variables when data splitting?

TL;DR: I don't want to include certain ID variables for the model creation, but I'm unsure if getting rid of those variables before the data split would create bias, especially in standarized event-based data.

What's the general rule of thumb for when to get rid of seemingly unimportant ID variables that were only used to create the relational data set? When is it better to get rid of them before the data split or after the data split?

Thank you.

Answer 1

Unique IDs (e.g. a match ID, when each record is a match) are unneccessary for either data splitting or model creation, so there's no harm removing them. There may be a benefit in removing them, to help prevent a learning algorithm "detecting" a spurious correlation.

For non-unique IDs (e.g. ones you have added to facilitate data source joins), if you include both the ID variable for a source and the attributes from that source, there will be a high degree of correlation between the key and that group of attributes. This could cause problems if you use a learning algorithm that assumes independence between the variables, such as linear/logistic regression.

As for keeping the ID variables for data splitting - you haven't included links to the posts you refer to so I'm not sure what point they are trying to make. Just including the ID variables in your data when creating the train/test splits isn't going to achieve anything, so I assume they are discussing using ID variables as part of the splitting strategy. Two possible strategies are stratified splitting (where the proportion of records with each value of the stratification variable is the same in the training and test sets) and group splitting (where records with the same value of the grouping variable are either all in the test set or all in the training set). If you want to use one of these splitting strategies you may want to use one of your ID variables for this. However, I suspect in most cases there would be a "real" attribute you could use instead. This would work just as well and would have the advantage of making it easier to explain, if you later need to explain your splitting strategy.