I'm training Bert on question answering (in Spanish) and i have a large context, only the context exceeds 512, the total question + context is 10k, i found that longformer is bert like for long document, but there's no pretrained in spanish so, is there any idea get around bert.
What i tried is:
from transformers import BertConfig config=BertConfig.from_pretrained(BERT_MODEL_PATH) config.max_length=4000 config.max_position_embeddings=4000 config.output_hidden_states=True model = MyBertModel(config)
but still gives me an error mismatch
RuntimeError: Error(s) in loading state_dict for BertModel: size mismatch for bert.embeddings.position_embeddings.weight: copying a param with shape torch.Size([512, 768]) from checkpoint, the shape in current model is torch.Size([4000, 768]).