Logistic Regression: Is it viable to use data that is outdated?

Question

TLDR: Want to predict who makes the playoffs (1,0), but there are more playoff spots now than there were in the past, is it okay to use that past data?

I want to use binary logistic regression on MLB data to estimate each team's probability of reaching the playoffs this upcoming season.

There is data going back as far as the seasons of the 1870s. However, my issue is that the structure of the playoffs and baseball as a whole has changed often over the years. Specifically, the changes deal with the number of playoff spots, which is in part due to an increase in the number of teams. For example, up until 1969 there were 20 teams, and there was only the championship (World Series), so, technically, only 2 teams made it to the "playoffs". The number of playoff spots has increased gradually to its present state, which is 10, in 2012, and there are now 30 teams.

To me, it makes sense to only use data from 2012 (to 2019) since it reflects the state of the upcoming season. This gives me 240 observations, thus 80 positive outcomes for my playoff (dependent) variable. However, I have about 40 predictors after removing highly correlated ones, which means that I should have way more observations. Though I know that the number of predictors will likely decrease once I fit the model, I still fear my sample size may still be too low. This makes me consider going further back to the previous era beginning in 1994 when there were 8 playoff spots, simply for the sake of more observations.

My question is that would it be viable to use such data in a regression, given that it may not accurately reflect the circumstances of what I'm trying to estimate? Could I maybe even go back to 1969?

I found this article which is pretty much exactly what I'm trying to do, and he uses data back to 1969, but it just seems like an issue to me.

Answer 1

Your thinking is sensible. Indeed, in a perfect world, your training data should be completely representative of the data you'll encounter. However, in practice, you often find that "unrepresentative" data may still have some value.

Ultimately, whatever you do is good if it improves your model, so if using "outdated" data helps, then do it!

Here is what you could experiment with:

• Let the data speak

You could compare models using more or fewer data and it might give you an idea of the ideal cutoff point.

• Implement "time decay"

Maybe, 1870 data is useful but likely, it's less useful than 1871 and even less useful than last year's. You could weight your training instance based on how old they are so that your recent data points have a bigger impact.

• Create time-insensitive features

By this I mean, you could find a way to turn your problem such that the number of playoffs team doesn't matter. Instead of a binary "playoff/no playoff", your problem could be to instead rank your teams, then you can select the playoffs team based on the cutoff at this specific year.

You could also add how many playoff teams there are as a feature so that the learning algorithm is aware of how many spots there are.