I have a system which sends invitations to users to participate in online questionnaires and want to use machine learning in order to predict the likelihood of fulfilling the questionnaires in a predefined time ( i.e. within 1 day , 2 day , 3 days ,a week, 2 weeks, etc) based on various feature related to the users to whom the invitations are sent to , details of the questionnaires ( i.e. how long are they , their topic ,etc.) , other contextual data ( time of day , day of year , in which media the invitation are sent - i.e. sms / email etc). I can train with positive examples ( invitations that were responded to by users ) and negative examples ( invitations that WERE not responded to ) , however , I'm not sure how to take into consideration the "predefined time" into the feature vector. for example, should I simply include a feature of "days since invitation was sent" and in the positive examples include the time , and in negative examples replicate each example X each of the predefined times to indicate that the users didn't respond at all ?

Any advise would be welcome !


You have a problem of data leakage. The "days since invitation was sent" feature contains all the information on the concept. Therefore, adding it as a feature will prevent most common classifiers from using rules based on other features and will lead to misleading results.

Duplicating the negative row for each predefined time value will lead to a different distribution than the one you should be tested one, which need needles problems.

Note that even that way, the feature might leak information (e.g., a small ratio of the positives will replay in two weeks while its distribution in the negatives will be higher, making it a good predicator for negative).

In case that you will have more complex representation of time (e.g., was the email sent on weekend? was the reply sent on weekend), the duplication of rows will be more complex.

I would try a different direction. It is likely that the negative rows has no influence on the time to reply in the positive ones. Therefore I would do a first research aimed to differ between replies and no replies without using the predefined time.

After that, do a second research only on the positives where the concept will be the predefined time, aiming to find what is influencing out given that the user had replied.

  • $\begingroup$ Thanks a lot Dan. so you're basically suggestion 2 classifiers - one for predicting response / no response , and the second one would incorporate the "days since invitation was sent" - how would you do that ? maybe the "days since invitation sent" response time should be actually the output of the classifier ( having for example labels per days ) ? Note that the purpose of the classifier would be to predict getting responses within X days. $\endgroup$ Mar 7 '17 at 13:49
  • $\begingroup$ in the second case, the "days since invitation was sent" should indeed be the concept - what you try to predict and not one of the features that you use. You might benefit from treating it as a regression problem (days to reply) and not a classification problem (will replay in x days). Then you will be able the predict response in X days by first predicting for a reply and then predicting them time to reply. $\endgroup$
    – DaL
    Mar 7 '17 at 14:38
  • $\begingroup$ Thanks Dan. If I treat it as a regression problem - how would I infer the probability of a reply with X days ? For example , suppose that for some input X , the second classifier returns 6 days. Now how would I infer from this the probability of responses with in 5 days ( simply use the proportions of the 5/6 ? ) and 12 days ( assuming 100% since 12 > 6 ) ? ( of course this all has to be multiplied by the probability of getting a response from the first classifier . $\endgroup$ Mar 7 '17 at 14:50
  • $\begingroup$ The answer to this question depends on the data. Taking 5/6 can be an estimation but usually the data doesn't behave like that. You can start by ignoring the features and just evaluate the distribution of days to return. This way you will have an estimation every period (which will be the same for all rows). You can then derive from the regression result adaptation to every period. Note that since this is not what you train the regression to do, you should check the results and ensure that they fit the data well enough. $\endgroup$
    – DaL
    Mar 8 '17 at 6:40
  • $\begingroup$ What if instead of regression I used classification ( the classes being the days ) and for estimating the probability to a reply with X days I would sum the probabilities returned by the classifier for each day i in {1,... ,X} ? $\endgroup$ Mar 8 '17 at 11:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.