I am fairly new to machine learning and data imputation so forgive me if I am using wrong words or not feasible ideas. I have a table as follows, with text in the Headline column and a list of keywords in the Keywords column:

Heading Keywords
Text [kwd, ..., kwd]
Text NaN
Text [kwd, ..., kwd]
... ...

My idea to impute the missing values in the keywords column is:

  1. Unpack the keywords list exploding the list of keywords with df.explode() - this will create multiple rows with the same text tough, not too sure if it's a good idea
  2. Apply text transformation to the headline column in this order - normalization, stemming, tokenization
  3. Apply TF-IDF to extract features from plain text
  4. Create the training data set splitting the data frame between non-missing values and missing values, where non-missing values data frame is used for training
  5. Use a machine learning model using the missing data text column as my X_test, where y_pred = model.predict(X_test)
  6. Use y_pred to substitute the missing entries in the keywords column

Now my questions are:

  1. Is this a correct "pipeline"? If not how would you advise to proceed?
  2. What model would suit best this application?
  3. I was thinking that step #1 can be substituted by a step at the end using a Multi-Output regressor, which means that I have to vectorize each word in the keywords list, maybe using one-hot encoding. Is this correct and a better solution?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.