2
$\begingroup$

I have a dataset where each response variable is the number of successes of N Bernoulli trials with N and p (the probability of success) being different for each observation. The goal is to train a model to predict p given the predictors. However observations with a small N will have a higher variance and higher N.

Consider the following scenario to illustrate better: Assume coins with different pictures on them have a different bias and that the bias is dependent on the picture on the coin. I have a large number of coins each with a different picture on them and each with a different bias p. I want to create a model that can predict the bias of a coin given only the picture on the coin. I flip each coin a different number of times and record the number of successes and total number of flips. So my data set consists of each picture and its estimate p=successes/flips.

So my question is when training my model how should I handle this. It seems more weight should be given to observations with a higher sample size(number of flips). I don't think it makes sense to include number flips as a predictor variable because the point is to build a model which predicts p using only the picture on the coin so this difference in variance for the response for each observation should be taken into account when training the model.

I am using several types of model but mainly working with keras and xgboost

$\endgroup$
  • $\begingroup$ Welcome to the site. How are you planning to predict this probability when every coin's probability is different and you have no common features between coins that you can use in numeric form? The number of trials for a single coin will only tell you how confident you are that your estimated p from N flips for that coin is in fact that coin's actual probability/bias (i.e., you can do hypothesis testing for each coin separately). It won't tell you anything about other coins. This seems analogous to trying to predict how many times a certain letter will appear in text only given that letter. $\endgroup$ – AlexK Apr 28 at 21:54
  • $\begingroup$ This is a contrived example to illustrate the problem. As I stated we assume that the bias of each coin is dependent on the picture on the coin. So the pictures on the coins are the common feature. A neural net could be used to create numeric features from the picture. My data set is not actually coins with pictures I just thought it might be easier to think of the problem in terms of coin flips. I hope this makes sense. $\endgroup$ – dln Apr 28 at 22:24
  • $\begingroup$ I may have confused the problem by using the example of coins with pictures on them, you could use anything e.g. the diameter, depth and weight of the coin for example for the predictors. $\endgroup$ – dln Apr 28 at 22:35
0
$\begingroup$

I may be understanding the question now. Still using the coin example, as I said above, the number of trials for a given coin only affects the confidence for the estimated probability/bias for that one coin. So it seems like you are asking how to incorporate that "confidence" into the response variable, if at all. In other words, you are asking if your model should reflect the uncertainty concerning the true value of $p$ for each coin, given the number of coin flips you performed.

I don't think assigning different weights to observations is appropriate in this situation because, again, the number of flips for one coin does not have anything to do with other coins.

I am not sure if this will satisfy your needs, but there is something called interval regression that is used to model a response/dependent variable that is defined as an interval between lower and upper bounds. It is a type of regression for censored data (a problem where the true value of response is not known) and is typically used for modeling such variables as income ranges or survival times. In your case, you could compute a confidence interval for the true value of $p$ for each coin, using the $p$ calculated from your trials and the number of trials specific to each coin. Then you would use this regression with two response variables: lower limit and upper limit of the confidence interval.

Based on my quick search, I am not finding a lot of Python support for this type of model, except:

Maximum Margin Interval Trees - Decision trees for interval regression

Drouin, A., Hocking, T.D. & Laviolette, F. (2017). Maximum Margin Interval Trees. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

https://arxiv.org/abs/1710.04234

https://aldro61.github.io/mmit/

$\endgroup$
  • $\begingroup$ What I have come up with is something along the lines of estimating the expected error due to chance for each example and subtracting it from the error using a custom loss function. so loss=mse-k where k is the estimated error due to chance. This is calculated based on the observed p and known n. Where p is the observed coin bias and n is the number of flips. So k will be higher when p is closer to 0.5. and k will be higher when n is low. $\endgroup$ – dln Apr 30 at 5:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.