0
$\begingroup$

I'm working on my custom Named Entity Recognition model that I'm making in Python's Keras lib. I have read that I should enumerate all words that are appearing, so that I get vectorized sequences. I have done that like this:

word2idx = {w: i + 1 for i, w in enumerate(words)}
label2idx = {t: i for i, t in enumerate(labels)}

# CREATING FEATURES(X) AND RESULTS(Y)
max_len = 50 
num_words = len(num_words) #number of unique words in dataset
X = [[word2idx[w[0]] for w in s] for s in list_of_sentances]
X = pad_sequences(maxlen=max_len, sequences=X, padding="post", value=num_words-1)

y = [[label2idx[w[1]] for w in s] for s in list_of_sentances]
y = pad_sequences(maxlen=max_len, sequences=y, padding="post", value=label2idx["O"])
y = [to_categorical(i, num_classes=num_labels) for i in y]

This is my final model:

input_word = Input(shape=(max_len,))

model = Embedding(input_dim = num_words, output_dim = 50, input_length = max_len)(input_word)
model = SpatialDropout1D(0.2)(model)
model = Bidirectional(LSTM(units = 5, return_sequences=True, recurrent_dropout = 0.1))(model)
out = TimeDistributed(Dense(num_labels, activation = "softmax"))(model)

model = Model(input_word, out)
model.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 30)]              0         
_________________________________________________________________
embedding (Embedding)        (None, 30, 50)            2187550   
_________________________________________________________________
spatial_dropout1d (SpatialDr (None, 30, 50)            0         
_________________________________________________________________
bidirectional (Bidirectional (None, 30, 10)            2240      
_________________________________________________________________
time_distributed (TimeDistri (None, 30, 11)            121       
=================================================================
Total params: 2,189,911  #LOOK AD THIS NUMBER
Trainable params: 2,189,911
Non-trainable params: 0

My accuracy is 98% and loss is 0.07. I like those results, but I have problem with making the prediction, because of the missing words. For example:

text = "I live in the Ohio and my name is Alex Wright and I work in AvcCC LTD"
text = text.split()
text = [word2idx[w] for w in text]

text = np.array(text)
print(text)
text=text.reshape(1,text.shape[0])

max_len = 50
text = pad_sequences(maxlen=max_len, sequences=text, padding="post", value=num_words-1)
print('PREDICTION')
res = model.predict(text).argmax(axis=-1)[0]
print(res)

ERROR:

KeyError: 'AvcCC'

In my dataset, and vocab there are no word 'AvcCC', how to handle that?

I want to use that code/model in production. Since my word2idx contains only words that were in starting data, how can I handle words that are not in my word2idx vocabulary? For example, its not possible for my word2idx vocabulary have all names and last names that exists, or all cities/locations, all company names, slang words etc.

My vocabulary had around 40k enumerated words (thats the number of unique words in my dataset). Then, I have enriched it with more than 100k other words. (I have made a web crawler that crawled different types of news articles). So now, my vocab has around 140k words. Now, instead of enumerating unique words from dataset, I'm loading my new word2idx/vocabulary.

word2idx = open('english-vocab.json')
word2idx = json.load(word2idx)

max_len = 50 
num_words = len(num_words) #number of unique words in dataset
X = [[word2idx[w[0]] for w in s] for s in list_of_sentances]
X = pad_sequences(maxlen=max_len, sequences=X, padding="post", value=num_words-1)

y = [[label2idx[w[1]] for w in s] for s in list_of_sentances]
y = pad_sequences(maxlen=max_len, sequences=y, padding="post", value=label2idx["O"])
y = [to_categorical(i, num_classes=num_labels) for i in y]

Accuracy and loss remained the same, but my model became much more slower because of the Total params (I can not use num_words anymore because it shows error, I need to use len(word2idx))

input_word = Input(shape=(max_len,))

model = Embedding(input_dim = len(word2idx), output_dim = 50, input_length = max_len)(input_word)
model = SpatialDropout1D(0.2)(model)
model = Bidirectional(LSTM(units = 5, return_sequences=True, recurrent_dropout = 0.1))(model)
out = TimeDistributed(Dense(num_labels, activation = "softmax"))(model)

model = Model(input_word, out)
model.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         [(None, 30)]              0         
_________________________________________________________________
embedding_1 (Embedding)      (None, 30, 50)            5596600   
_________________________________________________________________
spatial_dropout1d_1 (Spatial (None, 30, 50)            0         
_________________________________________________________________
bidirectional_1 (Bidirection (None, 30, 10)            2240      
_________________________________________________________________
time_distributed_1 (TimeDist (None, 30, 11)            121       
=================================================================
Total params: 7,598,961 # MUCH BIGGER NUMBER
Trainable params: 5,598,961
Non-trainable params: 0

With creating my own word2idx I wanted to handle missing words in vocab, but only thing I did is that I slowed down training of my model.

How can I handle this kind of problem? How to handle missing/non-existing/unknown words ?

$\endgroup$

2 Answers 2

0
$\begingroup$

Unknown words is an integral part of bringing NLP models to production. I recommend considering these methods:

  1. remove unknowns - the most trivial way to handle unknown words - just delete them. this is not optimal because of trivial reasons so let's continue.
  2. unknown tag - add new word to your vocabulary that represent unknown words, you can call it [UNK]. It is recommended to replace some rare words during training with [UNK] for robustness.
  3. characters - work in character level instead of word level.
  4. BPE - an algorithm that enables you to split words to subcomponents in a smart way, and leaves no unknown words. You get the best of both world - word level, characters level and now also subword level.

Each one of these methods has pros and cons and I recommend reading more about them.

$\endgroup$
0
$\begingroup$

There are various methods to handle words like usage of UNK, Char encoding, Subword encoding, Byte pair encoding. Let's see how to use each of them in following code snippets.

  1. Replace with UNK: To use this just add UNK or unknown word inside your vocab i.e. words list. Whenever you want to come across unrecognized word replace with UNK or unknown id.

e.g. text = [word2idx.get(w, word2idx.get('UNK')) for w in text]

  1. Subword encoding: It is little different than word2vec and glove which considers word as a smallest entity but algorithms like Fasttext brakes down rare word into sub words to calculate it's vector based on Skipgram and CBOW.

e.g. Check fasttext python implementation and play with it to get idea

  1. Byte pair encoding: BPE is slightly modified in its implementation such that the frequently occurring subword pairs are merged together instead of being replaced by another byte to enable compression. BPE is slightly modified in its implementation such that the frequently occurring subword pairs are merged together instead of being replaced by another byte to enable compression.

e.g. Check huggingface library for it's implementation as Transformers uses BPE.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.