Machine learning data leakage

Question

I have a dataset with Information about a transaction that a customer made in the store. the target is Indication of whether fraud was committed in the transaction.

One of my feature is SuspiciousWords_Score. The meaning of this feature is how many suspicious words appeared in the invoice of that transaction

Suspicious words are determined by the number of times the word appeared in a fraudulent transaction in the last 2 years.

Of course, for some of the records, words are counted that at that point in time the word that appeared in the invoice was not suspicious because the word did not appear in the invoices of transactions where there was fraud before that transaction, but since the pool of suspicious words was determined when the dataset was created, then this word is counted for that transaction.

My question is - is this an data leakage or not?

Answer 1

Yes, this is likely data leakage: the features contain information which was obtained from the target variable, including the target found in the test.

From the point of view of evaluation, this clearly breaks the principle that the test set cannot be used before evaluation. For this problem the solution would be to exclude the test set from the stage of calculating the set of "suspicious words" (btw this case is equivalent to feature normalization).

But a more appropriate method would to consider the time factor: the set of "suspicious words" could be calculated at different points in time, and for every instance the feature SuspiciousWords_Score would be calculated based on the latest set at the time of the transaction.