I am working on a model where the underlying data is inherently correlated by groups. So some of my observations are almost duplicates but not quite.

The problem is pretty simple, I have a y variable to predict from a discrete x variable and several other potential predictor variables which may or may not be significant. The observations are not quite independent, they're taken from groups of underlying events but I want to handle this better.

I could approach the problem by only selecting one observation from each underlying group but I think this would lose a lot of data and statistical power. I would like some method where I can keep all my datapoints but possibly impose some artificial weight scheme or something to weight data less if there are near-duplicates elsewhere or if they are from the same underlying event.

In particular, I'm using a Bayesian regression approach with pymc3 so could do some fudging of variances in my likelihood to fit less for values which have lots of close duplicates from the same underlying group.

Does anyone have experience with similar problems and solutions to deal with it?
I think my model is currently very much overfitting due to high correlations between some predictors and predictions.

  • $\begingroup$ What you describe is EXACTLY Bayesian hierarchical/mixed effects modeling. No need for ad-hoc approaches, Bayesian hierarchical modeling using group id as a random effect (or equivalently, the effects of each group comes from a common distribution with some mean) is a very natural approach. $\endgroup$
    – aranglol
    Mar 9 at 18:53

Some models are fine with duplicate data. For example, this is how oversampling works in unbalanced datasets, Also Naive Bayes should handle this. However there are many other models will end up giving high prediction rates because of this. Duplicate rows can cause problems due to the same data being in the train and test datasets. Those rows will give a high accuracy to the model, since the training dataset already knows the answer. Exact duplicates can be found and removed using


Finding similar data will be a bit more difficult. You can use these same 2 functions on a partial set of columns if you think that will find more matches. Or you can run the data through a cluster algorithm (k-means) and sample from the different clusters, or add a weight column based on the clustering output, maybe divide by the number of points in each cluster. You can then pass in this weight vector into the model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.