I am a bit confused on this matter, I can't find any resources that touch on the following but my logic says that embeddings do introduce data leakage in time series:

Considering a temporal dataset that describes match outcomes like this:

date player1 player2 elo1 elo2 score1 score2 winner
2015-01-01 Curry Rose 1500 1500 5 2 1
2015-01-01 Nobody Wade 1500 1500 0 10 0
2015-01-01 Nobody Wade 1450 1550 0 8 0
2016-01-01 Varejao Lebron 1400 1450 3 5 0
2016-01-01 Varejao Rose 1350 1600 3 7 0

Where player columns describe the matchup, elo1/2 is a rating for each player (which updates after each match), score1/2 is the match score and winner is the target variable.

If I were to make a ML model to predict winners, then I need to account for features not known at inference time. Meaning Elo is a rating that is known beforehand, but scores are only known after a match is being played. So I could use players, elos as is but I'd need to use an average up to time t-1 for a match at time t (e.g. average of last 5 matches) for scores. Split datasets into train/val/test etc. Now if I were to use embeddings for the player columns, my question is:

Since each player will be encoded with a single vector for all times t, won't that introduce data leakage? To illustrate this, imagine that all players start with a starting Elo of 1500, but some later in their careers become superstars (very high elo) -- and seen in training data. This means that the fresher Lebron, in the example, will be represented by the same vector as the superstar Lebron. So his current superstar Elo influences his fresher matchups. Isn't that data leakage? Or is it assumed that the temporal order is somehow coded in the vector?

If the above does not introduce data leakage, how is it different from training embeddings on the dataset as is (so using end game scores) and the transferring these embeddings to a model that respects feature data leakage?? What is the difference?

Thanks for any pointers or input

  • $\begingroup$ Why would each player will be encoded with a single vector for all times t? Why not have these vectors be time-dependent? $\endgroup$
    – Cryo
    Apr 6 at 19:36
  • $\begingroup$ Each player will encoded by a single vector because the 'vocab size' will be the number of unique players. Can you expand more on the time-dependent suggestion? Do you mean to have a dynamic timeframe where each player is considered more than once? e.g. x number of players * y timeframes, so x*y input dimension? $\endgroup$ Apr 7 at 20:34
  • $\begingroup$ Will your single vector contain information on the elo1/2 of the player? If so, and since you yourself said that elo will change (don't actually know what that is), the vector has to be time-dependent. How did you actually want to create your vectors to represent players? What information goes in there? If it is some emergent embedding of capacity to win, then probably prior N games, with scores, and outcomes, and prior elo-s should be the inputs for your embedding encoder, which you would then train to create an embedding that best allows to predict the outcome of the next match. $\endgroup$
    – Cryo
    Apr 7 at 21:39
  • $\begingroup$ You could then use the encoder part on its own to update the embedding vector with fresh elo-s $\endgroup$
    – Cryo
    Apr 7 at 21:39
  • $\begingroup$ I'm very confused by your reply. Elo is a ranking system where the numeric representation quantifies a player's skill. These values are known beforehand, but update for the next match. I am talking about the simplest example where player1,2 are passed in an embedding layer, elo1,2 and average_score1_L5, average_score2_L5 is the rest of the input with winner being the target. My main question is whether the embedding vectors are considered to leak data $\endgroup$ Apr 8 at 9:49


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.