I'm working on a project that aims to classify JIRA issues into their relevant owner group.
An issue has the following text features:
all of which are text based.
During prediction time, however, no comments are available as the goal of the tool is to predict the owner without having the users to assign it back and forth between them.
I wonder if that's ok to use the comments only for the training dataset but completely ignore it on the test dataset.
Is that considers data leakage? I'm having a bad feeling it may be so.
I've asked a very similar question a while back and I was told I can benefit the comments data during training as it will enrich the vocabulary.
Needless to say, I'm getting far better results when using them (in both datasets)
I was thinking maybe it is ok to split the dataset into train and test and dropping the
comments column from the train dataset
Is it ok to use it or am I just cheating myself?
ADDING SOME CLARIFICATION:
The 3rd feature (AKA comments) as well as the other ones eventually being united into 1 column called
text so it's pretty much having this "extra" text during training VS not having it during prediction time