0
$\begingroup$

For GLM: General Language Model Pretraining with Autoregressive Blank Infilling ,

  1. May I ask how is the sampling for input division in step (b) being done ?

  2. why in step (c), the green x3 is moved to the end ? why is the maximum value in Position 1 limited to 5 instead of 6 ?

  3. why Part A tokens cannot attend to Part B tokens ? but Part B tokens can attend to A ?

GLM

$\endgroup$

1 Answer 1

0
$\begingroup$
  1. The sampling strategy is described in Section 2.1. Specifically, we design three different sampling strategies for different types of downstream tasks.
  2. The order of [x5, x6] and [x3] is randomly shuffled. Position 1 represents the position in the corrupted text and there are only 5 tokens in the corrupted text
  3. The model needs to predict the tokens in Part B. If tokens in Part A can attend to tokens in Part B, the model can obtain the information from the representations of Part A.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.