0
$\begingroup$

I'm working on a project that aims to classify JIRA issues into their relevant owner group.

An issue has the following text features:

  1. Summary
  2. Description
  3. Comments

all of which are text based.

During prediction time, however, no comments are available as the goal of the tool is to predict the owner without having the users to assign it back and forth between them.

I wonder if that's ok to use the comments only for the training dataset but completely ignore it on the test dataset.

Is that considers data leakage? I'm having a bad feeling it may be so.

I've asked a very similar question a while back and I was told I can benefit the comments data during training as it will enrich the vocabulary.

Needless to say, I'm getting far better results when using them (in both datasets)

I was thinking maybe it is ok to split the dataset into train and test and dropping the comments column from the train dataset

Is it ok to use it or am I just cheating myself?

ADDING SOME CLARIFICATION:

The 3rd feature (AKA comments) as well as the other ones eventually being united into 1 column called text so it's pretty much having this "extra" text during training VS not having it during prediction time

$\endgroup$

1 Answer 1

0
$\begingroup$

Using the comments doesn't really make sense if they will not be available for prediction. If you add the feature you will end up having to impute the comment features in some way. How do you plan on doing this? Also when evaluating your model, you should also discard the comments and impute the missing values to get realistic evaluation metrics. I am pretty sure adding them will add no predictive power to your model, the only thing that could happen is overfitting.

$\endgroup$
9
  • $\begingroup$ I guess you mean "will not be available for prediction". but how about gaining from their data to enrich the vocabulary (during EDA)? $\endgroup$
    – Ben
    Mar 8, 2023 at 22:48
  • $\begingroup$ yes sorry. Editing the typo. $\endgroup$ Mar 8, 2023 at 22:50
  • $\begingroup$ I've updated my question as I think I didn't explain my use case good enough - please have a 2nd look $\endgroup$
    – Ben
    Mar 8, 2023 at 22:54
  • $\begingroup$ it was clear. I really wouldn't do that. I think it's very counter-intuitive and prone to error. Technically, as long as the way you evaluate your model is rigorous and the comments are never used for validation or testing, you could do it. But I think it should not help in any way if you actually do it properly, and will just add noise that will prevent your model from capturing whatever little signal will come from Summary and Description. Comments will have many more tokens than the other sections $\endgroup$ Mar 8, 2023 at 22:56
  • $\begingroup$ here's a link to my older somewhat similar question: datascience.stackexchange.com/questions/96945/… You can see the answer I'm getting is somewhat confusing as opposed to yours $\endgroup$
    – Ben
    Mar 8, 2023 at 22:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.