3
$\begingroup$

I'm working on a prediction project where we have a lot cyclical features such as hour of the day, weekday, month, day of year, etc etc. After some searching I decided to follow the advice here.

Now I have the sin and cos component for every cyclical feature as a separate feature, so month becomes month_sin and month_cos. However, I don't know for sure whether the model can deal with this correlation, as both components need to be equally weighted in order for the feature to make sense. The model assigns different weights to the sin and cos components after training though. My intuition tells me that this is bad, but I'm not sure what to do about it.

Currently gbm (R) gives the best results. For a gradient boosting model, is it better to force equal weights on the two correlated features, or is it better to let the model figure it out even if it results in different weights on the two components? Or would you suggest an entirely different approach?

$\endgroup$
4
$\begingroup$

as both components need to be equally weighted in order for the feature to make sense

That is not the case.

For instance if $\text{sin}(\theta)$ of the cyclical feature is weighted strongly, it means that the original feature has the strongest positive effect on output at $\theta = \frac{\pi}{2}$.

If the two features are weighted equally, then the focus is around $\theta = \frac{\pi}{4}$

In general, you should expect different weights to apply to the different components, depending on which hours of the day maximise the target variable and which minimise it.

For a gradient boosting model, is it better to force equal weights on the two correlated features, or is it better to let the model figure it out even if it results in different weights on the two components?

Definitely better to let the model figure it out in this case.

Your main concern would be whether a one-hot-encoded representation might be better than a cyclical representation for your problem. One-hot allows for arbitrary relationships to each hour, but adds more dimensions - and so may require more examples of each time. Cyclical has less dimensions, but is more likely to include non-linear effects, if for instance min and max effect on output target variable are not exactly 12 hours apart - thus it may require a more complex model.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your response! I considered one-hot testing and decided against it, since it would disregard information about proximity of values (e.g. January and February are closer than January and June, wouldn't one-hot encoding discard this?) $\endgroup$ – Armen Jul 18 '18 at 11:25
  • 1
    $\begingroup$ @Armen: Yes one-hot coding will lose the ordering information. It is possible that it is not as important as you think though. It depends on the problem. $\endgroup$ – Neil Slater Jul 18 '18 at 12:33
  • $\begingroup$ Can't we also try categorical entity embeddings? @Neil $\endgroup$ – Aditya Jul 18 '18 at 17:09
  • 2
    $\begingroup$ @Aditya: You could, but for a simple cyclic value such as day of week or hour of day I would of thought that would be overkill (mainly because the sin/cos mapping already captures the core simple relationship between categories). Might depend if you could re-use an embedding from another task which had a lot more data but a similar relationship to the problem. I'd expect embeddings to be a half-way house between one-hot and sin/cos features, in that you could have more that 2 dimensions, but less than e.g. 24 for hours of the day. $\endgroup$ – Neil Slater Jul 18 '18 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.