1
$\begingroup$

I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable?

Thanks in advance,

$\endgroup$
  • 1
    $\begingroup$ Hi, welcome to Data Science SE! The answer will surely depend on your case. Please explain further your case, share part of the data if possible, and explain why you think year should be used as a categorical variable. $\endgroup$ – Romain Reboulleau Nov 26 '19 at 10:55
  • $\begingroup$ Hi, is data related to Real Estate listings. So I have listings dated from the 1500s to the 2019s and even more. My thought is that I don't know if it's advisable to use "year" as a numerical variable due to the fact that it's not actually a number but a year, and for me, it seems to be categorical but rather ordinal, as years are ordered. $\endgroup$ – Luis Nov 26 '19 at 11:25
1
$\begingroup$

First of all I would suggest you to analyze the data. Do you really need the year? Have you studied the correlation with the other variables? I mean, study the domain of the model you intend to create before create any model.

You can treat year as categorical variable and use some of the techniques such as One Hot Ecoding or Dummy Variables for better performance. You could also perform a normalization of the years, treating them as numerical variables, which are between 0 and 1.

The most important thing is to know what we are going to do to make good use of the information or data.

$\endgroup$
  • $\begingroup$ Hi Sergio, my problem is that I have tons of years, from 1500s to 2019s, so turning it into a Dummy Variable might be really highly-dimensional. It is Real Estate data, so the year can be a determinant on variables like prices, so in my opinion, I would need it. How would you proceed? $\endgroup$ – Luis Nov 26 '19 at 11:27
1
$\begingroup$

Year or indeed any time dimension is a hard thing to include into a ML model because it begs one question:

Are we looking at a time series or not?

Time series behave categorically different than data that is not ordered sequentially and we have to model them differently.

You could treat year/time like any other dimension and use it as a predictor in a regression-based model. But that would not be ideal!

Instead try to use models that are suited to time series analysis like ARIMA or even deep learning models like LTSTM.

A straight-forward way to treat this would be to model a simple prediction based on a time-series model like ARIMA and then build a second model on top that takes in all the other predictors and tries to predict the residuals from the time series model.

$\endgroup$
1
$\begingroup$

Your problem statement is not really clear but from what I have read seems that you could use Year as the index in order to do time series validation splitting. You can read a bit about it here and use the implementations of sktime by Alan Turing Institute or classic scikit learn.

This will allow you to evaluate the performance of your model in a production environment since given the nature of your problem (real state) you will want to predict the prices for next given year.

So, rather than using it as a feature, indexing by it and then splitting.

If you are interested in using it as a feature you can calculate elapsed time from the first year and this will help the model to see if there is a trend over time. Or as categorical and do target encodings.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.