Hill climbing with noisy data

Question

For example

• Shoes priced at \$10, 30 of 1000 customers buy for revenue of \$300
• Shoes priced at \$20, 25 of 1000 customers buy for revenue of \$500

And then now I have the problem of determining how certain I am the second pricing is yields higher returns

My question is: How can I find out X so that I'm X% confident that the \$20 price yields higher returns than the \$10?

Answer 1

When in doubt, bootstrap.

• Make a list with 970 0s (the non-sales at \$10) and 30 10s (the sales at \$10) - customers_10
• Make a list with 975 0s (the non-sales at \$20) and 25 20s (the sales at \$20) - customers_20
• Repeat lots of times (maybe N = 100,000 times - increase this until the the result settles down)
  • Randomly take 1000 samples (with replacement) from both lists - samples_10, samples_20
  • Sum the samples for each list - sum_s10, sum_s20
  • Record which one generated more revenue (i.e. whether sum_20 > sum_10)
• The percentage of times sum_s20 was greater than sum_s10 is your X

(Another approach, which would be equivalent in the limit of N large, would be to model this as a binomial process. But processing is cheap, bootstrapping is robust and easy to implement - getting the binomial model right would be easier to mess up).