New to data science and I am working on a personal project in sports analytics. I have data in the form of one dataset for each year/season.

Each dataset has the team's wins, losses, and various offensive and defensive stats. These will be pandas.DataFrames indexed by team name.

My goal is to predict wins and losses based on this data for each year, and then create a more generalized model to predict next season using all of the previous years data.

I am having difficulty deciding how best to combine the data. I think that it would be very inefficient to combine all of the datasets into 1 and have variables for each stat in the year prior, 2 years prior, etc. How is this type of sequential data usually handled?

(For clarification, I plan on using gradient boosting via xgboost package and an LSTM via the Keras package)


Generally, to perform machine learning all data needs to be in a single dataframe. Team name, better yet team id, should be the primary key. Then modeling each year requires selecting the relevant columns (e.g., wins_year_1, loses_year_1, …). Finally, the general model would select all the columns.

It is best practice to transform all data before modeling. This avoids making transformation errors on some of the data which can lead to errors in modeling.

Pandas support this type of combining through the merge command.

Typically, this data is not called sequential data. Sequential data implies a notion of state. The state changes values over time. In most sports, recent performance is not a strong indicator of how likely a team will the next game. This is often referred to as the Hot Hand Fallacy. Thus, current "win/lose" state of a team is not a strong feature in modeling.

  • $\begingroup$ Thanks for the answer. One other question, since the current state is not a good indicator of success, would a more effective way to model be to use the various stats simply as more examples? In other words, determine how strongly each stat correlates with wins/losses and points scored, rather than treating it as a time series model? $\endgroup$ – agn Aug 7 '17 at 15:44
  • $\begingroup$ Yes! In Machine Learning that process is called feature engineering, the process of finding combinations of features that are related to predicting the variable of interest. $\endgroup$ – Brian Spiering Aug 8 '17 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.