Building BERT tokenizer with custom data

Question

I'm wondering if there is a way to train our own Bert tokenizer instead of using pre-trained tokenizer provided by huggingface?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. — Community, Jun 20 at 13:43

noe · Accepted Answer · 2023-06-20 14:22:46Z

0

Yes, of course, it is possible, but that implies that you need to train again the whole model, not only the tokenizer.

Here you can find the steps to do everything with the HuggingFace libraries, including a complete example.

answered Jun 20 at 14:22

noe

23.8k1 gold badge45 silver badges73 bronze badges

$\begingroup$ Thanks a lot! I think I got what I want. $\endgroup$
– Loius Leong
Jun 20 at 17:36

Add a comment |

1 Answer 1