We have a click-model which is currently being used for search ranking in production, and I want to create a new model which takes the old model click probability as one input and adds some other variables in too. The problem is that training data will positionally biased by the fact that the probability of a click is correlated to the old model's prediction.
My plan is to introduce a penalty factor on the original model's prediction to ensure that it doesn't dominate the new model (eyeballing the results to decide on an appropriate penalty factor). Is this approach valid or would there be a better way to approach this?
Note that I don't want to rebuild the old model with the new variables because
- The existing model takes a long time (days) to build
- The new model and old model will be deployed separately, i.e. the old model will be scored offline/batch whereas the new model will be scored real-time