0
$\begingroup$

While training XLMRobertaForSequenceClassification:

xlm_r_model(input_ids = X_train_batch_input_ids
            , attention_mask = X_train_batch_attention_mask
            , return_dict = False
           )

I faced following error:

Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py", line 1218, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py", line 849, in forward
    past_key_values_length=past_key_values_length,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py", line 132, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/sparse.py", line 160, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 2044, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

Below are details:

  1. Creating model

    config = XLMRobertaConfig() 
    config.output_hidden_states = False
    xlm_r_model = XLMRobertaForSequenceClassification(config=config)
    xlm_r_model.to(device) # device is device(type='cpu')
    
  2. Tokenizer

    xlmr_tokenizer = XLMRobertaTokenizer.from_pretrained('xlm-roberta-large')
    
    MAX_TWEET_LEN = 402
    
    >>> df_1000.info() # describing a data frame I have pre populated
    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 1000 entries, 29639 to 44633
    Data columns (total 2 columns):
    #    Column  Non-Null Count  Dtype 
    ---  ------  --------------  ----- 
    0    text    1000 non-null   object
    1    class   1000 non-null   int64 
    dtypes: int64(1), object(1)
    memory usage: 55.7+ KB
    
    X_train = xlmr_tokenizer(list(df_1000[:800].text), padding=True, max_length=MAX_TWEET_LEN+5, truncation=True) # +5: a head room for special tokens / separators
    >>> list(map(len,X_train['input_ids']))  # why its 105? shouldn't it be MAX_TWEET_LEN+5 = 407?
    [105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, ...]
    
    >>> type(train_index) # describing (for clarity) training fold indices I pre populated
    <class 'numpy.ndarray'>
    
    >>> train_index.size 
    640
    
    X_train_fold_input_ids = np.array(X_train['input_ids'])[train_index]
    X_train_fold_attention_mask = np.array(X_train['attention_mask'])[train_index]
    
    >>> i # batch id
    0
    >>> batch_size
    16
    
    X_train_batch_input_ids = X_train_fold_input_ids[i:i+batch_size]
    X_train_batch_input_ids = torch.tensor(X_train_batch_input_ids,dtype=torch.long).to(device)
    
    X_train_batch_attention_mask = X_train_fold_attention_mask[i:i+batch_size]
    X_train_batch_attention_mask = torch.tensor(X_train_batch_attention_mask,dtype=torch.long).to(device)
    
    >>> X_train_batch_input_ids.size()
    torch.Size([16, 105]) # why 105? Shouldnt this be MAX_TWEET_LEN+5 = 407?
    
    >>> X_train_batch_attention_mask.size()
    torch.Size([16, 105]) # why 105? Shouldnt this be MAX_TWEET_LEN+5 = 407?
    

After this I make the call xlm_r_model(...) as stated at the beginning of this question and ending up with the specified error.

Noticing all these details, I am still not able to get why I am getting the specified error. Where I am doing it wrong?

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.