Should I create a separate column for each Id value in a feature column or can I use the feature column as it is?

Question

I am working on developing a model for predicting, revenue that a movie will make. One of the features in the training set contains id of the series that a movie belongs to.Say, Star Wars series has Id 1, then the corresponding value in that column for "Return Of Jedi" movie will be 1.

Should I create a separate column for each of the series and add value 1 if the film belongs to that series, or will just keeping the existing column containing ids will be fine?

Answer 1

You should not leave the column as it is since similar id values are not semantically related. Maybe a deep neural network could handle this, but linear models definitely do not.

There are two options, depending on the cardinality (the number of unique entries/categories) of the column.

1. If the cardinality is low you should one-hot-encode the series, one column for each series, like you propose. However, in your dataset, there are probably many series, and your data will end up having too many features. See this image (source), you need one column/feature for each category if you one-hot encode.

2. If the cardinality is high, you should find an encoding for the id column. One way would be using target statistics: Calculate the mean target (in your case revenue) a movie of a series makes for each series-category of the training set. Then replace the id column with a column of the target-means for the respective series. If there is a series in the test data, which is not in the training data, just use the overall target mean of the training set. It is important for this approach that you never use target-statistics calculated from the test set, otherwise, your error estimate will be highly optimistic! The rationale behind this approach is, that the model is allowed to learn things like "continuations of successful movies are often successful".

Answer 2

Yes, create a "dummy" (or indicator) variable (1,0) for each ID. This is sometimes called a "fixed effect", indicating the "identity" of each ID (in this case the movie series). So Star Wars can be expected to generate very different revenue than other films. In a linear model, the dummy will simply introduce a own intercept coefficient for each ID. So you can think of this as allowing Star Wars to have a structurally different turnover than other movie series.

There are many options to dummy-encode single columns in a DF, so that it os no problem to work with dummies/indicators. E.g. use model.matrix in R:

mydf = data.frame(c(1,1,2,2))
colnames(mydf)<-c("X1")
mydf$X1 <- as.factor(mydf$X1)
mydf

mymat = model.matrix(~ X1 -1 , data=mydf)
mymat

This will recode mydf

  X1
1  1
2  1
3  2
4  2

To look like:

  X11 X12
1   1   0
2   1   0
3   0   1
4   0   1

It is easy to illustrate how "dummies" work in a linear regression. Say your model looks like (ommiting subscripts $i$ for convenience):

$$y = \beta_0 + \beta_1 x + u,$$

where $\beta_0$ is the intercept, $x$ is a continuous feature, and $u$ is the error term. If you have two "IDs" (like Star Wars yes or no), you introduce an additional variable =1 if Star Wars and =0 otherwise. Call this vector $I$.

Now your model looks like:

$$y = \beta_0 + \beta_1 x + \beta_2 I + u.$$

When you predict a non-Star Wars film ($I=0$) you would do:

$$ \hat{y} = \hat{\beta_0} + x * \hat{\beta_1} + 0 * \hat{\beta_2}. $$

In case of a Star Wars film you would do:

$$ \hat{y} = \hat{\beta_0} + x * \hat{\beta_1} + 1 * \hat{\beta_2}. $$

So in this case the intercept term is $\hat{\beta_0} + \hat{\beta_2}$ and $\hat{\beta_2}$ is just what makes Star Wars different from the other film(s) in this simple illustration.