1
$\begingroup$

Say you have a binomial distribution with $p$ very small ($\approx 0.001$).

You are asked to predict the conditional success rate $SR=S/T$ with $S$ successes out of $T$ trials given a set of conditions $X$.

One would expect (correct me if I'm wrong, though I ran simulations and am quite confident) $SR$ to have a downward with increasing $T$ when $T$ is still small, and to approach $p$ when $T$ is large.

The distribution of trials and successes is not uniform across $X$, so the train set tends to give higher $SR$ to conditions with smaller $T$ values.

How would you treat this data skew when constructing a model?

Improve this question
$\endgroup$
1
  • $\begingroup$ I'm voting to close this question as off-topic because its a statistics question and it would belong better on stats.stackexchange.com $\endgroup$
    – Spacedman
    Commented Apr 1, 2016 at 17:32

1 Answer 1

Reset to default
1
$\begingroup$

When fitting a GLM (at least in R), I know there is a optional weight vector that you can include. This weight is not to give more importance to an observation, but to rather weight observations based on $T$ for example.

The R documentation says:

 For a binomial GLM prior weights are used to give the number of trials
 when the response is the proportion of successes

So I believe this would help adjust for your non-uniformity. How to choose those weights can be tricky I suppose and can vary a lot with your data, but its worth looking into if it looks like it will help your problem.

Simulation of problem

Do you have enough data where you can play with dropping data, or maybe if you simulate how much the $SR$ is inflated at low $T$ you can try to adjust for it somehow. It looks like after about 1000 trials the high success rate issue begins to mellow out, and after 3000 or so you start getting more reliable measures.

Improve this answer
$\endgroup$
4
  • $\begingroup$ While I understand that this is the common practice in such cases, I don't think it affects the prediction as I would like. It will 'ignore', or allow more freedom in the space where weights are small, instead of actually correcting the skew. $\endgroup$
    – scf
    Commented Mar 1, 2016 at 19:06
  • $\begingroup$ Ah I see what you mean, added some more stuff that maybe someone else can add on to. I'm not sure what else to try, but let us know what you end up doing! $\endgroup$
    – TBSRounder
    Commented Mar 2, 2016 at 15:11
  • $\begingroup$ My simulations showed that it mellows out at ~ 50,000 trials. This problem stems from the fact that for small p: prob(S<pT) > prob(S>pT), so now I'm thinking perhaps I should normalize the distribution as a nonlinear function of T. I have loads of data, but still I'd rather compensate the lacking conditions than drop data from the abundant ones. $\endgroup$
    – scf
    Commented Mar 2, 2016 at 17:53
  • $\begingroup$ @scf you are wrong about the reason. The median for your p is exactly round(pT), so it fluctuates between being larger or smaller than pT at regular intervals en.wikipedia.org/wiki/Binomial_distribution#Median. The distribution of S/T is slightly skewed because for small p, S is closer to Poisson. But what you really see decreasing, is the variance of S/T, so you might want to include a variance term in your model. This is called regularization. $\endgroup$
    – Valentas
    Commented Apr 1, 2016 at 16:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.