I am working on a supervised learning problem for a web-search task, where I have access to a relatively small set of human-labeled examples and lots of user-behavior data.

Now, user behavior data is biased, because of presentation bias, position bias etc. So it's likely that its' distribution will be different from human-labeled data.

I am planning to use both to train a Neural Network model.

Now I am confused about how to combine both datasets?

  • $\begingroup$ nice topic, thanks for asking about it; I am not sure about understanding the bias due to the user behaviour, can you give any more detail on it? $\endgroup$
    – German C M
    Aug 11, 2020 at 7:18
  • $\begingroup$ Thanks for responding. So the task is query classification into some predefined categories. From user clicked documents, can create a dataset, but it will be biased by documents on the top position, plus the mode of presentation etc. $\endgroup$ Aug 11, 2020 at 13:09
  • $\begingroup$ Could you provide some samples and/or the structure of your data $\endgroup$ Aug 17, 2020 at 13:51
  • $\begingroup$ I think you are looking for semi-supervised methods. $\endgroup$ Aug 17, 2020 at 16:32
  • 1
    $\begingroup$ @EngrStudent I am not sure if it falls under the semi-supervised umbrella. We have two kinds of labelled data. One of them is human-labelled, the other one is derived from the click-log. In semi-supervised learning, you have lots of unlabelled data. $\endgroup$ Aug 18, 2020 at 6:49

1 Answer 1


That is a common scenario in a learning to rank problem. One heuristic is to separately model explicit (human-labeled) and implicit (user-behavior) features. Then combine the separate feature groups with a learned weight for their final relative contribution. Improving Web Search Ranking by Incorporating User Behavior Information by Agichtein et al goes into greater detail.

RankNet takes this approach using a neural network.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.