I'm wondering if there is a way to train our own Bert tokenizer instead of using pre-trained tokenizer provided by huggingface?
1
-
$\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$– Community BotCommented Jun 20, 2023 at 13:43
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
Yes, of course, it is possible, but that implies that you need to train again the whole model, not only the tokenizer.
Here you can find the steps to do everything with the HuggingFace libraries, including a complete example.
-
$\begingroup$ Thanks a lot! I think I got what I want. $\endgroup$ Commented Jun 20, 2023 at 17:36