0
$\begingroup$

I need help in deciding whether my below implementation imposes data snooping bias and information leakage from the test/evaluation set to the train set.

I have a text corpus of 10k+ short online comments. Many have special symbols and emojis. These might theoretically be handled by vectorizers (by transforming them to their Unicode representation for instance) but I believe it's more valuable to transform them to their descriptions because that would provide (at least with a higher likelihood) actual information on whether the user is happy, sad, frustrated, etc. Thus my function I have written for this purpose applies the following transformation:

original = "hello this is a sample comment 😇"
transformed = "hello this is a sample comment SMILING FACE WITH HALO EMOJI"

My plan is to train and evaluate multiple models using cross validation and hyperparameter tuning, for binary classification. At first, I just loaded my entire dataset (i.e. before the train/test split) and used df.apply(lambda x: transform_emojis(x)) where transform_emojis() is the function that performs the above string transformation.

However, I am now wondering if this might be a mistake, and whether I'd better incorporate this function into the pipeline, right before the vectorization step. Let's consider an example: a very rare emoji (e.g. squid: 🦑) might be present in only one comment out of the 10k+. If in one iteration of the K-fold cross validation the squid happens to show up in the test set, the count vectorizer (or tf-idf) fitted on the training set would know nothing about this symbol and drop it during the .transform() of the test set.

In contrast, my current implementation transforms the entire corpus before modelling. Continuing with the squid example this means that irrespective of whether the squid shows up in the training or test set, it is already converted to string representation: "SQUID". If it shows up in the training set, the vectorizer will then create a separate feature named 'squid'. If it shows up in the test set, the vectorized will again not know anything about it and hence the word 'squid' will be thrown away during the .transform() step.

It seems to me that there is no risk of data leakage and data snooping bias in this case but I am somewhat uncertain whether my reasoning is correct.

$\endgroup$

1 Answer 1

1
$\begingroup$

While technically this may be a situation of information leakage since you're applying a pre-processing step on the whole dataset before performing a dataset split, I don't think it matters too much in this specific case. Normally these types of steps would have to be done after the data is split to make sure that there is no information leaking to the training dataset that is not in the training dataset itself (e.g. new keywords). However, since the mapping from emoji to words is already known beforehand (i.e. is not derived from information in any new data but is static) and the number of emoji's isn't likely to change (to my basic understanding at least) I wouldn't expect it to have any noticeable impact. I would still include this step in the pipeline though because it would make the overall process cleaner and it shouldn't impact the results given that the mapping is static.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.