I have additional variables in my dataset that are somewhat correlated to the continuous target variable, but that are completely unavailable in the test set. So, I'm wondering how the best to use that variables.
- The first approach I come up with, is to add these variables to the target variables, perhaps, giving more weight to the real target variable in the loss function. The disadvantage of this approach is that it unnecessary complicates the model and it's not actually trained for what it's needed.
- The other approach is to add them to the input vector with additional column
'IsNanVar'for each unavailable variable that indicates if the variable is present. Then, we can generate data for training randomly replacing unavailable columns to median and turning
Truewith some probability. The disadvantage of this approach is that test data aren't distributed like the training data and input vector is unnecessary long.
In both cases, I use cat2vec technics to learn encodings for categorical variables while training. So, my idea is that we can gradually decrease either weights for unneeded targets in the loss in the first case, or ratio of unavailable columns in the second case, eventually remaining with zero weights for unneeded targets, or train set turning to look completely like test set. So, my idea is that categorical embeddings would contain this additional info about unused variables. In the end, it's possible use this for kind of transfer learning, replace either last or first layer and retrain the model with input and output needed for test set.
So, I'm wondering is there simpler approach, or any other kind of approach to use here?
My question differs from the following one: How to handle features which are not always available? My features are completely unavailable in test set.