Training on derived features that won't be present in a test set

Question

So I have an odd real-world problem in which the data that's going to be fed to a categorical predictive model has (a few) certain features, and when building a model I can bolt on additional datasets that add context to some of those features, but when it's going to be running live I won't be able to use any of that additional data - the model itself needs to already "know" what's important about specific values of the original data and be able to operate just on those attributes.

The major features here are time(/date/DOW) and location (lat/long) within a city. So, the additional datasets are context about that location (zoning, demographics, transit) or time (traffic, commercial activity, etc). But for performance reasons (join cost, etc) we can't use them live, we only have the time and location numbers (plus easily parse-able derivatives).

I'm imagining a process whereby the model uses those additional features to learn which subsets or demarcations of time-and-location actually matter, and can then continue to use those weightings and rules or whatever even in the absence of the additional features, just relying on the numerical values. But it's also possible that there's no such approach. Can anyone suggest an algorithm or approach that could let us retain some of that value?

Answer 1

It sounds like a great opportunity for feature engineering. You're contemplating this in your last paragraph so you're on the right track, but I'll elaborate on a possible solution here.

You could use the features that you know will exist in the test set to construct synthetic features based on the context information. For example, you could predict the traffic given the datetime and location, whereas zoning would only be based on location. Similar with demographics and transit. Your goal at this stage would be to construct an algorithm to build each synthetic feature, knowing you will have to apply it per row on the test data (and any incoming data upon deployment).

Then train your algorithm using the features that will exist in the test set and the synthetic features (not the extra context information).

After that, construct the synthetic features in the test data. You're not creating new data here, you're just transforming features into what will hopefully be more descriptive features. Now you can use the real and synthetic features to evaluate the model you trained previously. I'm not saying this will definitely give you better results, but it's something I would explore.

Ultimately this is just giving your ML algorithm a head start. Instead of blindly training a generic algorithm on the features that exist, we can use logic and common sense (and some extra information) to build part of the algorithm first (the feature transformations) to give the final classifier a better chance. Additionally, this way you can encode the information from the extra training data without having to deal with missing features in your test set.

Answer 2

Thats not how it works, there is not juice to be extracted if the data is missing from the test set.

You will have features in train that might be discriminative, but missing in test. Then when you try to predict there wont be any inputs that you can map to the outputs.

This beeing said I should mention that NaiveBayesclassifier will not use missing features, so there would be no harm in including them but you wont get any additional information, just more unnecessary complexity.

Answer 3

Though most of the Libraries will through an error for this, even if we manage(let's assume) to create the features based on any logic e.g. Fill NAN, it will not work

ML model just creates a pattern based on data. If it has created a pattern using some set of features and at the moment of prediction those features are unavailable it will definitely impact the result.

I believe we are trying to solve a data engineering problem in the ML space. We must see why can't we resolve the join/cost issue.