We survey our members on a regular basis on various topics, but with always the same question (basically, do you like it? yes or no), and we classify them based on what is replied, and how many participated in the survey.

When we send the email, we would like to be able to know at what point in time we can estimate that the final result (how many people will fill, what % of yes) is likely to be the one we have now.

We did process the previous surveys, and we can see a pattern where after 6 or 12 hours, the result is similar to the final one (ie when we wait more than a week after the mailing). Obviously, a survey sent friday at 8pm takes longer to reach a conclusive stage than the ones sent on tue at 9am. They are other factors too, for instance if our mail provider experienced delay delivering, then of course we need to wait longer.

Is there a way to know how long to wait (3, 6, 12... hours?) before we get a solid prediction of the result?

Is there an algorithm that can predict the result at any point in time and give an estimate on how likely the estimate is correct (p value like)?


Think about this problem in two parts.

Part 1) How many samples do you need in order to have the desired confidence in the Null hypothesis?

Part 2) How long does it take to get that many samples?

Part 1 can be answered with A-priori power analysis. Use this technique to determine the sample size needed to get the p-value desired.

Part 2 can be addressed with an algorithm applied to the data you have collected thus far. Your predictors in this case would be:

Column 1: the probability that your statistic of interest has leveled off (is not changing significantly with additional input).

Column 2: date/day/time start. You probably will need to codify this.

Column 3: number of samples at finish. These should be the number which you determined from your prior analysis.

Your response/output in this case would be:

Column 4: the date/day/time finish.

Of course would need to do some data analysis to determine the appropriate algorithm to use on this data set. The end product should be a function that takes as input a target probability, date/day/time start and a target sample size, while outputting a date/day/time finish.

I hope this at least gives you some ideas on how to proceed.

| improve this answer | |
  • $\begingroup$ Thanks, and very useful to split in two parts indeed. Let me play with my data a bit more, but you unlocked the situation and put me in the right track, thanks! $\endgroup$ – a25bedc5-3d09-41b8-82fb-ea6c353d75ae Oct 21 '16 at 7:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.