I am working on developing a model for predicting, revenue that a movie will make. One of the features in the training set contains id of the series that a movie belongs to.Say, Star Wars series has Id 1, then the corresponding value in that column for "Return Of Jedi" movie will be 1.

Should I create a separate column for each of the series and add value 1 if the film belongs to that series, or will just keeping the existing column containing ids will be fine?


You should not leave the column as it is since similar id values are not semantically related. Maybe a deep neural network could handle this, but linear models definitely do not.

There are two options, depending on the cardinality (the number of unique entries/categories) of the column.

  1. If the cardinality is low you should one-hot-encode the series, one column for each series, like you propose. However, in your dataset, there are probably many series, and your data will end up having too many features. See this image (source), you need one column/feature for each category if you one-hot encode.

  2. If the cardinality is high, you should find an encoding for the id column. One way would be using target statistics: Calculate the mean target (in your case revenue) a movie of a series makes for each series-category of the training set. Then replace the id column with a column of the target-means for the respective series. If there is a series in the test data, which is not in the training data, just use the overall target mean of the training set. It is important for this approach that you never use target-statistics calculated from the test set, otherwise, your error estimate will be highly optimistic! The rationale behind this approach is, that the model is allowed to learn things like "continuations of successful movies are often successful".

  • $\begingroup$ Even if the cardinality is high and I use one hot encode, then also, I will have only 1 column right? Just the value in the columns will be very large. Each value in the column can be like '1000','00001'. Where each position will represent a movie series. $\endgroup$
    – V K
    Jan 11 '20 at 5:47
  • 1
    $\begingroup$ No, certainly not. If you one-hot encode a categorical feature you need to create a column for each category of the feature. This column will only contain 0 or 1 in each row. What you propose would be just giving a different name to the series. I update the answer with an example of a one-hot encoding $\endgroup$
    – PascalIv
    Jan 11 '20 at 14:09
  • $\begingroup$ Also, for the films which do not belong to any series, will it be fine if I assign same id for all of them? $\endgroup$
    – V K
    Jan 13 '20 at 5:34
  • 1
    $\begingroup$ Yes! That would be the way to go. $\endgroup$
    – PascalIv
    Jan 13 '20 at 10:00

Yes, create a "dummy" (or indicator) variable (1,0) for each ID. This is sometimes called a "fixed effect", indicating the "identity" of each ID (in this case the movie series). So Star Wars can be expected to generate very different revenue than other films. In a linear model, the dummy will simply introduce a own intercept coefficient for each ID. So you can think of this as allowing Star Wars to have a structurally different turnover than other movie series.

There are many options to dummy-encode single columns in a DF, so that it os no problem to work with dummies/indicators. E.g. use model.matrix in R:

mydf = data.frame(c(1,1,2,2))
mydf$X1 <- as.factor(mydf$X1)

mymat = model.matrix(~ X1 -1 , data=mydf)

This will recode mydf

1  1
2  1
3  2
4  2

To look like:

  X11 X12
1   1   0
2   1   0
3   0   1
4   0   1

It is easy to illustrate how "dummies" work in a linear regression. Say your model looks like (ommiting subscripts $i$ for convenience):

$$y = \beta_0 + \beta_1 x + u,$$

where $\beta_0$ is the intercept, $x$ is a continuous feature, and $u$ is the error term. If you have two "IDs" (like Star Wars yes or no), you introduce an additional variable =1 if Star Wars and =0 otherwise. Call this vector $I$.

Now your model looks like:

$$y = \beta_0 + \beta_1 x + \beta_2 I + u.$$

When you predict a non-Star Wars film ($I=0$) you would do:

$$ \hat{y} = \hat{\beta_0} + x * \hat{\beta_1} + 0 * \hat{\beta_2}. $$

In case of a Star Wars film you would do:

$$ \hat{y} = \hat{\beta_0} + x * \hat{\beta_1} + 1 * \hat{\beta_2}. $$

So in this case the intercept term is $\hat{\beta_0} + \hat{\beta_2}$ and $\hat{\beta_2}$ is just what makes Star Wars different from the other film(s) in this simple illustration.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.