I have a dataset D:

X = D.drop(columns=['target'])
y = D['target']

D is large, but contains huge number of duplicates - and I want to speedup the learning process. I can't simply drop these duplicates, because this will lead to bias in the distribution. But I can do the following thing:

  1. Compute duplicate counts and save it separately as SW.
  2. Drop duplicates from the dataset and save it as D1.


X1 = D1.drop(columns=['target'])
y1 = D1['target']

Now, I can estimate some metric on pre-trained model like this:

score(y1, model.predict(X1), sample_weight=SW)

this will be exactly equal to (tested for roc_auc_score):

score(y, model.predict(X))

Similary I can pass SW as sample_weight parameter to model.fit(...) function. model.fit(X, y) will not be equal to model.fit(X1, y1, sample_weight=SW) - but let it be.

The problem is - how do I can properly perform train_test_split of X1, y1, and SW? For example:

X               y
feature         1  
feature         1
feature         1
feature         1
feature         1

after duplicate removal it will be:

X1              y1        SW
feature         1         5

after train_test_split(test_size=0.2) I expect:

X1_train      y1_train    SW_train
feature       1           4

X1_test       y1_test     SW_test
feature       1           1

If this is not possible in Sklearn, is there any ML library, that provides this functionality?


1 Answer 1


The most straight forward way to solve your problem is to split the data set before you replace duplicates with weights, and then replace the duplicates with weights only in the training set. This maintains the idea that replacing duplicates with proportional sample weight is an implementation detail in the training process where everything you can do to minimize the size of the data matters.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.