How to handle multiple datasets of sequential data

Question

New to data science and I am working on a personal project in sports analytics. I have data in the form of one dataset for each year/season.

Each dataset has the team's wins, losses, and various offensive and defensive stats. These will be pandas.DataFrames indexed by team name.

My goal is to predict wins and losses based on this data for each year, and then create a more generalized model to predict next season using all of the previous years data.

I am having difficulty deciding how best to combine the data. I think that it would be very inefficient to combine all of the datasets into 1 and have variables for each stat in the year prior, 2 years prior, etc. How is this type of sequential data usually handled?

(For clarification, I plan on using gradient boosting via xgboost package and an LSTM via the Keras package)

Answer 1

Generally, to perform machine learning all data needs to be in a single dataframe. Team name, better yet team id, should be the primary key. Then modeling each year requires selecting the relevant columns (e.g., wins_year_1, loses_year_1, …). Finally, the general model would select all the columns.

It is best practice to transform all data before modeling. This avoids making transformation errors on some of the data which can lead to errors in modeling.

Pandas support this type of combining through the merge command.

Typically, this data is not called sequential data. Sequential data implies a notion of state. The state changes values over time. In most sports, recent performance is not a strong indicator of how likely a team will the next game. This is often referred to as the Hot Hand Fallacy. Thus, current "win/lose" state of a team is not a strong feature in modeling.