I have a dataset of form text, text, category, category, time, text and I would like to apply the attention mechanism to it. This requires that all inputs be in the same vector space. I am using a particular encoding method (from BERT) for the text-type data and I can build a custom trainable embedding for the category features. However, I don't have a good way of embedding time data.

Currently, my time feature is normalised on [0,1], and represents when over the time period (one year) the post was created. Naively, I would split this up into month, day of week, and time of day features to do feature engineering, but I don't have a good way to embed this in very high dimensional space (say, 500+ dimensional space).

What's the best option here? I would like to avoid tiling or repeating the same feature set to reach the requisite dimensionality -- is there a better way? I could put a trainable embedding layer on top of those three features, too, but this seems suboptimal.


I don't know if it's a good technique or not, but I saw this paper last year that proposes a time2vec, analogous to doc2vec, et al:

S. M. Kazemi et al., “Time2Vec: Learning a Vector Representation of Time,” arXiv:1907.05321 [cs], Jul. 2019.


It depends on what information you want to capture :

  • If you want to capture the passing of time, encoding your date as days since a reference date might be a good idea.

  • If you want to encode the cyclicality of time (months in a year), you can encode your month variables on a circle.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.