How do I encode time in high dimensional space?

Question

I have a dataset of form text, text, category, category, time, text and I would like to apply the attention mechanism to it. This requires that all inputs be in the same vector space. I am using a particular encoding method (from BERT) for the text-type data and I can build a custom trainable embedding for the category features. However, I don't have a good way of embedding time data.

Currently, my time feature is normalised on [0,1], and represents when over the time period (one year) the post was created. Naively, I would split this up into month, day of week, and time of day features to do feature engineering, but I don't have a good way to embed this in very high dimensional space (say, 500+ dimensional space).

What's the best option here? I would like to avoid tiling or repeating the same feature set to reach the requisite dimensionality -- is there a better way? I could put a trainable embedding layer on top of those three features, too, but this seems suboptimal.

Answer 1

I don't know if it's a good technique or not, but I saw this paper last year that proposes a time2vec, analogous to doc2vec, et al:

S. M. Kazemi et al., “Time2Vec: Learning a Vector Representation of Time,” arXiv:1907.05321 [cs], Jul. 2019.

Answer 2

It depends on what information you want to capture :

• If you want to capture the passing of time, encoding your date as days since a reference date might be a good idea.

• If you want to encode the cyclicality of time (months in a year), you can encode your month variables on a circle.