I want to train a model to detect wrong word using in sentence.
- I have 1 million sentences(word base or char base) with different length. Each position(word or char) has a label to indicate it is wrong or correct .
- If a sentence length is 10, I want LSTM learn 2 words surrounding current word ( so each word become 5 word) . So I need transform the sentence to [10, 5] .
- But the first word(doesn't have left words) and right word (doesn't have right words) , so I need pad them.
- Then use word2vec transform this sentence to [10, 5, embedding_size] , same as other sentences. (i.e each word transform to 5 word embeddings)
- trainning (feed model by [5, embedding_size] * bacth_size )
I saw a lot LSTM input format and finally work out above flow. Is my steps right ?
Actually I am still confusing with LSTM input format.
I have seen
- Some pad all sentence to same length
- Some split sentence into word blocks with same size (like I do above).
- others ....
I am trying to use code to generate error samples, and mix with correct samples for lstm training.
For example:
corrects = [
['How', 'are', 'you', '!'], # [1, 1, 1, 1]
['Fine', ',', 'thank', 'you', '.'], # [1, 1, 1, 1, 1]
['Do', 'you', 'have', 'meal', '?'], # [1, 1, 1, 1, 1]
...
]
wrongs = [
['How', 'were', 'you', '!'], # [1, 0, 1, 1]
['Find', ',', 'thank', 'you', '.'], # [0, 1, 1, 1, 1]
['Did', 'you', 'have', 'meal', '.'], # [0, 1, 1, 1, 0]
...
]
I come up with several thoughts
Because lstm can learn current word's network weight with pre words. So directly pass sentence to lstm model, don't transform from
[sentence_len, embedding_size]
to[sentence_len, step_len, embedding_size]
(step_len
means present a word with it surround words).Previous sentence's last word may be affacted by latter sentence's first few word, so It better chain all sentence . (How to BiLSTM in this situation ?)
Use feature like
[sentence_len, step_len, embedding_size]
, doesn't needBiLSTM
any more ? I think it still help here.
There are too many thoughts in my brain, hard to make it clear. So I post question here.