I need help in deciding whether my below implementation imposes data snooping bias and information leakage from the test/evaluation set to the train set.
I have a text corpus of 10k+ short online comments. Many have special symbols and emojis. These might theoretically be handled by vectorizers (by transforming them to their Unicode representation for instance) but I believe it's more valuable to transform them to their descriptions because that would provide (at least with a higher likelihood) actual information on whether the user is happy, sad, frustrated, etc. Thus my function I have written for this purpose applies the following transformation:
original = "hello this is a sample comment 😇"
transformed = "hello this is a sample comment SMILING FACE WITH HALO EMOJI"
My plan is to train and evaluate multiple models using cross validation and hyperparameter tuning, for binary classification. At first, I just loaded my entire dataset (i.e. before the train/test split) and used df.apply(lambda x: transform_emojis(x)) where transform_emojis() is the function that performs the above string transformation.
However, I am now wondering if this might be a mistake, and whether I'd better incorporate this function into the pipeline, right before the vectorization step. Let's consider an example: a very rare emoji (e.g. squid: 🦑) might be present in only one comment out of the 10k+. If in one iteration of the K-fold cross validation the squid happens to show up in the test set, the count vectorizer (or tf-idf) fitted on the training set would know nothing about this symbol and drop it during the .transform() of the test set.
In contrast, my current implementation transforms the entire corpus before modelling. Continuing with the squid example this means that irrespective of whether the squid shows up in the training or test set, it is already converted to string representation: "SQUID". If it shows up in the training set, the vectorizer will then create a separate feature named 'squid'. If it shows up in the test set, the vectorized will again not know anything about it and hence the word 'squid' will be thrown away during the .transform() step.
It seems to me that there is no risk of data leakage and data snooping bias in this case but I am somewhat uncertain whether my reasoning is correct.
