May I ask how is the sampling for input division in step (b) being done ?
why in step (c), the green
x3is moved to the end ? why is the maximum value in
Position 1limited to 5 instead of 6 ?
why Part A tokens cannot attend to Part B tokens ? but Part B tokens can attend to A ?
- The sampling strategy is described in Section 2.1. Specifically, we design three different sampling strategies for different types of downstream tasks.
- The order of [x5, x6] and [x3] is randomly shuffled. Position 1 represents the position in the corrupted text and there are only 5 tokens in the corrupted text
- The model needs to predict the tokens in Part B. If tokens in Part A can attend to tokens in Part B, the model can obtain the information from the representations of Part A.