# sklearn text analysis - dealing with missing values

I'm working on a multi-class text classification project.

My goal is simple: given a "bug", I'd like to predict to which final group owner it will be assigned to.

I was able to achieve ~15000 samples of bugs with a relevant status (i.e. a status I can implicitly deduce the label ==> group owner from)

My feature list:

1. Subject
2. Description
3. Comments <== can I use it at all (there's a good chance it won't be available during prediction stage)?
4. Errors (being extracted via regex from the above features)
5. Panics (being extracted via regex from the above features)
6. Has images (boolean field) ==> ('yes' or 'no')
7. Involved groups ==> Not sure I can use it as it being derived from the comments
8. Committer groups
9. Reporter group
10. Assignee group

My labels is the final group name.

I've 6 different classes/labels

At the moment I'm getting score of ~80%

I guess my questions are:

1. Can I use features that are being contained/derived from a different ones (e.g. errors being derived from comments/description)?
2. How should I deal with missing features? For example Panics and Errors and not always exist, should I omit this feature? should I replace it with boolean feature such as (has errors - yes/no)?
3. For Committer groups I'm using the existing values to predict the new ones and add them to the dataframe, but while I can somehow understand how it can be done for such feature, I cannot see it being done to others.

1. Yes, extracting new features from existing ones is a common concept - one piece of the feature engineering process.
2. There are different methods to handle missing values. In general, you don't want to lose any available information. There are different imputation techniques for missing values that have different efficiency depends on: the model you used, feature type (categorical, numeric), number of different values, etc.
3. Not sure what you meant; you need to give more details.

good luck

• Regarding #3 - I've mentioned that I used a ML approach (prediction) to complete the missing values - I'm using the existing values to predict the missing ones. What I meant is that in this specific case, it's relatively achievable since the values range are group labels (number of classes). Is there some technic I can apply to complete/predict missing values for simple "free text" like the other columns (e.g. subject/description)?
– Ben
Jul 6, 2021 at 13:36
• @Kulikr I would like to mention that your point 2. does not apply to the SO question. I am assuming techniques you are talking about apply to data which is string or numerical. Those techniques do not apply to text data which the So clearly mentions in his question Dec 10, 2021 at 6:01