2
$\begingroup$

I am dealing with a data set of transcribed call center data, where customers are being recorded when interacting with the agent. This is then automatically transcribed by an external transcription system. I want to automatically assess the quality of these transcriptions.

Sadly, the quality seems to be disastrous. In some cases it's little more than gibberish, often due to different dialects the machine is not able to handle. We have no access to the original recordings (data privacy), so there is no way whatsoever to get or create the true labels. The system cannot be replaced as we are committed to it.

Again to the question: is there any way to automatically assess the quality of transcriptions with NLP methods? We want to quantify and compare transcription quality to filter out the best samples for semantic inference of our customers' input in a downstream task. I am thinking about something like a coherence measure in order to find the sentences which make the most sense, grammatically or semantically. Sadly, things as BLEU, WER or Rouge do not work in this case.

I'd be grateful for anything pointing in the right direction. Most importantly again, we have no labels and it needs to be scalable.

Thanks a lot!

$\endgroup$
2
$\begingroup$

There is at least one way:

  1. Create/Acquire a grammar model for the language spoken (there are several such models for various languages used in NLP)
  2. Test the transcripts for beign grammaticaly/syntacticaly correct.
  3. This assesment will at least rule out gibberish and most of transcripts that do not correspond to valid sentences of the language spoken
$\endgroup$
5
  • $\begingroup$ Do grammar models also work if the transcriptions don't have punctuations? (we cannot separate the plain text into sentences). Thank you! $\endgroup$
    – hoang tran
    Jan 26 at 8:38
  • 1
    $\begingroup$ as far as i know the grammar models can be adjusted to work without explicit punctuation present $\endgroup$
    – Nikos M.
    Jan 26 at 8:45
  • $\begingroup$ Could you please name a few grammar models? It would be great if you could also include some links. Thank you very much! $\endgroup$
    – hoang tran
    Jan 26 at 8:51
  • 1
    $\begingroup$ I dont have links to such models handy and it depnds on language spoken, but would be easy to search online or even construct one from scratch $\endgroup$
    – Nikos M.
    Jan 26 at 8:53
  • 1
    $\begingroup$ a quick search gave me this; nlp.stanford.edu/software/lex-parser.shtml $\endgroup$
    – Nikos M.
    Jan 26 at 8:56
0
$\begingroup$

I have recently worked on the ASR system of speech to text using a bunch of dialects and the current state of the art says that the best way to handle dialects is using the approach of XLSR, fine-tuning your model to recognize a dialect while you are pre-trained it on an N number of language. To do so and build such a system, is using Transformers where actually you can assess your WER and PER without even perceive how data looks like or even understand the language in the first place. Because dialect is not often an easy way to understand in case we are not native speakers.

That said, I just wanna help in the regard of how to handle well your model with dialects and you can check this out it helped me a lot: https://arxiv.org/abs/2006.11477 and also you can check this https://arxiv.org/abs/2006.13979.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.