I am new to building models for Speech ASR. I want to build a model for Speech Transcription in Urdu. I used XLS-R (following the example by Patric von Platen on Huggingface Colab notebook. My notebook can be found) as it is pre-trained on Urdu too. I trained the model on Common Voice Urdu dataset, but the Word Error Rate (WER) is 50%, which is bad. I want to improve it by using a better decoder model, as suggested in the original tutorial notebook.

My question is:

  • For my problem, which decoder models can I use? XLS-R is a good speech encoder model as it is pre-trained in 128 languages including Urdu, which model or models should work well with it?
  • Should I build a decoder model from scratch, or should I use a pre-trained decoder model and fine-tune it? What role does CTC compatibility play in it?

Any help or guidance would be appreciated, or any link to a notebook of someone working on similar problem. Thank you.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.