I have a dataset of form
text, text, category, category, time, text and I would like to apply the attention mechanism to it. This requires that all inputs be in the same vector space. I am using a particular encoding method (from BERT) for the
text-type data and I can build a custom trainable embedding for the
category features. However, I don't have a good way of embedding time data.
Currently, my time feature is normalised on
[0,1], and represents when over the time period (one year) the post was created. Naively, I would split this up into
day of week, and
time of day features to do feature engineering, but I don't have a good way to embed this in very high dimensional space (say, 500+ dimensional space).
What's the best option here? I would like to avoid tiling or repeating the same feature set to reach the requisite dimensionality -- is there a better way? I could put a trainable embedding layer on top of those three features, too, but this seems suboptimal.