# Bert for QuestionAnswering input exceeds 512

I'm training Bert on question answering (in Spanish) and i have a large context, only the context exceeds 512, the total question + context is 10k, i found that longformer is bert like for long document, but there's no pretrained in spanish so, is there any idea get around bert.

What i tried is:

from transformers import BertConfig
config=BertConfig.from_pretrained(BERT_MODEL_PATH)
config.max_length=4000
config.max_position_embeddings=4000
config.output_hidden_states=True
model = MyBertModel(config)



but still gives me an error mismatch

RuntimeError: Error(s) in loading state_dict for BertModel: size mismatch for bert.embeddings.position_embeddings.weight: copying a param with shape torch.Size([512, 768]) from checkpoint, the shape in current model is torch.Size([4000, 768]).

• QA is not my field of expertise, so I recommend you take a look at the QA papers in paperswithcode (link)