I'm new to machine learning, being this the first time I'm involved in a project in the area. I have a dataset of news articles and have extracted the keywords present on the news title such as ['china factory activity shrinks', 'first time', '2 years']. Note, the list size varies.

I would like to use this data in my features, but I don't know how and what is the best way to extract features based on a list of keywords.

Since the keywords cannot be converted to a categorical value, I think I can't use OneHotEnconder, but I'm not 100% certain. Is bag-of-words or TF-IDF viable possiblity? How can I encode this keywods to numerical values maintaining some meaningful information to the model?

  • $\begingroup$ You may use Word2Vec to vectorize the tokens. $\endgroup$ Apr 18, 2020 at 2:23
  • $\begingroup$ @ShubhamPanchal I have looked to word2Vec, but I'm just wondering what application it may have to the model, besides the word similarity. $\endgroup$
    – B.Crz
    Apr 18, 2020 at 14:29

1 Answer 1


Sure, of course we can use TF-IDF or bag of words. The easiest way is to build a separate TFIDF transformer for every group of keywords and then combine them together using FeatureUnion. Just make sure the custom tokenizer only filters out the keywords in each group that you're looking at.

from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "china factory activity shrinks some text here",
    "first time some text here",
    "2 years some text here",
    "combining first time and 2 years here.",
    "adding china factory activity first but doesn't match exactly"

# custom tokenizers

keywords1 = lambda x: [x for x in x.split() if x in ["china", "factory", "activity", "shrinks"]]
keywords2 = lambda x: [x for x in x.split() if x in ["first", "time"]]
keywords3 = lambda x: [x for x in x.split() if x in ["2", "years"]]

combined_features = FeatureUnion([
    ('kwd1', TfidfVectorizer(tokenizer=keywords1)),
    ('kwd2', TfidfVectorizer(tokenizer=keywords2)),
    ('kwd3', TfidfVectorizer(tokenizer=keywords3))

pipeline = Pipeline([('bow', combined_features)])
output_corpus = pipeline.fit_transform(corpus)

For the deep-learning approach, I would recommend a similar kind of pattern so that the embedding learned from each group would have a different semantic meaning from another group. You'll just need to ensure that the tokenizer only considers the keywords from each group when it is processed. If you're using a pre-build word embedding; you'll just have a filter before it goes into each embedding so that it doesn't "leak" into another topic.

Example using Keras:

from keras import layers
from keras.models import Model
from keras.preprocessing.sequence import pad_sequences

def custom_sequence_tokenizer(txt, vocab):
    vocab_mapper = dict([(word, idx+1) for idx, word in enumerate(vocab)])
    text_split = [vocab_mapper.get(x, 0) for x in txt.split() if x in vocab]
    return text_split

keywords1 = pad_sequences([custom_sequence_tokenizer(x, ["china", "factory", "activity", "shrinks"]) for x in corpus], padding='post', maxlen=10)
keywords2 = pad_sequences([custom_sequence_tokenizer(x, ["first", "time"]) for x in corpus], padding='post', maxlen=10)
keywords3 = pad_sequences([custom_sequence_tokenizer(x, ["2", "years"]) for x in corpus], padding='post', maxlen=10)

input_count1 = layers.Input(shape=(10,), name='in1')
input_count2 = layers.Input(shape=(10,), name='in2')
input_count3 = layers.Input(shape=(10,), name='in3')

# input size is reflection of the vocab size
embed1 = layers.Embedding(5, 8)(input_count1)
embed2 = layers.Embedding(3, 8)(input_count2)
embed3 = layers.Embedding(3, 8)(input_count3)

combine = layers.Concatenate()([embed1, embed2, embed3])

model = Model(inputs=[input_count1, input_count2, input_count3], outputs=combine)
output = model.predict({
    'in1': keywords1,
    'in2': keywords2,
    'in3': keywords3,
  • $\begingroup$ Thank you! I'll be looking to this then. $\endgroup$
    – B.Crz
    Apr 18, 2020 at 14:27
  • $\begingroup$ what is a recomended method I should follow to encode the result from the TF-IDF scores from the group of keywords? $\endgroup$
    – B.Crz
    Apr 20, 2020 at 18:03
  • $\begingroup$ That's really up to you and the domain tbh. You could just us a naive bag-of-words per group of keywords to start off with and depending on how that goes expand from there $\endgroup$
    – chappers
    Apr 20, 2020 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.