What is the proper train data format in LSTM?

Question

I want to train a model to detect wrong word using in sentence.

1. I have 1 million sentences(word base or char base) with different length. Each position(word or char) has a label to indicate it is wrong or correct .
2. If a sentence length is 10, I want LSTM learn 2 words surrounding current word ( so each word become 5 word) . So I need transform the sentence to [10, 5] .
3. But the first word(doesn't have left words) and right word (doesn't have right words) , so I need pad them.
4. Then use word2vec transform this sentence to [10, 5, embedding_size] , same as other sentences. (i.e each word transform to 5 word embeddings)
5. trainning (feed model by [5, embedding_size] * bacth_size )

I saw a lot LSTM input format and finally work out above flow. Is my steps right ?

---

Actually I am still confusing with LSTM input format.

I have seen

1. Some pad all sentence to same length
2. Some split sentence into word blocks with same size (like I do above).
3. others ....

I am trying to use code to generate error samples, and mix with correct samples for lstm training.

For example:

corrects  = [
    ['How', 'are', 'you', '!'],          # [1, 1, 1, 1]
    ['Fine', ',', 'thank', 'you', '.'],   #  [1, 1, 1, 1, 1]
    ['Do', 'you', 'have', 'meal', '?'],   #  [1, 1, 1, 1, 1]
    ...
]


wrongs  = [
    ['How', 'were', 'you', '!'],            # [1, 0, 1, 1]
    ['Find', ',', 'thank', 'you', '.'],    # [0, 1, 1, 1, 1]
    ['Did', 'you', 'have', 'meal', '.'],    #  [0, 1, 1, 1, 0]
    ...
]

I come up with several thoughts

1. Because lstm can learn current word's network weight with pre words. So directly pass sentence to lstm model, don't transform from [sentence_len, embedding_size] to [sentence_len, step_len, embedding_size] (step_len means present a word with it surround words).

2. Previous sentence's last word may be affacted by latter sentence's first few word, so It better chain all sentence . (How to BiLSTM in this situation ?)

3. Use feature like [sentence_len, step_len, embedding_size], doesn't need BiLSTM any more ? I think it still help here.

There are too many thoughts in my brain, hard to make it clear. So I post question here.

Answer 1

[sentence_len, embedding_size] to [sentence_len, step_len, embedding_size] (step_len means present a word with it surround words).

Use step_len=1, just the current word. after squeezing the tensor, its shape will be [batch_size,sentence_len,embedding_size].

Previous sentence's last word may be affected by latter sentence's first few words.

Use padding, append the last 3 to 5 words of previous sentences at the start of the current sentences, do not backpropagate for the errors of those padded words, by masking their loss. as for the case of first sentences of paragraphs, padded with a simple <pad> token.

doesn't need BiLSTM anymore? I think it still helps here.

Using BiLSTM is a must. if you want to limit step_len to be 1, by the way, you are computing the conditional probability of Y being true (correct/not correct) conditioned on the whole context (sentence). as you are feeding the sentence at once, the loss will be the summation over all single losses.

Side note:
To batch the training examples, you also need to pad the end of sentences with the three to five words of the next ones. use <pad> for sentences ending the paragraph, don't permit their contributions to the model (adjusting the weight), mask your padded examples to have zero loss. because their labels are just noise.