I have a dataset of form text, text, category, category, time, text
and I would like to apply the attention mechanism to it. This requires that all inputs be in the same vector space. I am using a particular encoding method (from BERT) for the text
-type data and I can build a custom trainable embedding for the category
features. However, I don't have a good way of embedding time data.
Currently, my time feature is normalised on [0,1]
, and represents when over the time period (one year) the post was created. Naively, I would split this up into month
, day of week
, and time of day
features to do feature engineering, but I don't have a good way to embed this in very high dimensional space (say, 500+ dimensional space).
What's the best option here? I would like to avoid tiling or repeating the same feature set to reach the requisite dimensionality -- is there a better way? I could put a trainable embedding layer on top of those three features, too, but this seems suboptimal.