2
$\begingroup$

I am dealing with a data set of transcribed call center data, where customers are being recorded when interacting with the agent. This is then automatically transcribed by an external transcription system. I want to automatically assess the quality of these transcriptions.

Sadly, the quality seems to be disastrous. In some cases it's little more than gibberish, often due to different dialects the machine is not able to handle. We have no access to the original recordings (data privacy), so there is no way whatsoever to get or create the true labels. The system cannot be replaced as we are committed to it.

Again to the question: is there any way to automatically assess the quality of transcriptions with NLP methods? We want to quantify and compare transcription quality to filter out the best samples for semantic inference of our customers' input in a downstream task. I am thinking about something like a coherence measure in order to find the sentences which make the most sense, grammatically or semantically. Sadly, things as BLEU, WER or Rouge do not work in this case.

I'd be grateful for anything pointing in the right direction. Most importantly again, we have no labels and it needs to be scalable.

Thanks a lot!

$\endgroup$
2
$\begingroup$

There is at least one way:

  1. Create/Acquire a grammar model for the language spoken (there are several such models for various languages used in NLP)
  2. Test the transcripts for beign grammaticaly/syntacticaly correct.
  3. This assesment will at least rule out gibberish and most of transcripts that do not correspond to valid sentences of the language spoken
$\endgroup$
5
  • $\begingroup$ Do grammar models also work if the transcriptions don't have punctuations? (we cannot separate the plain text into sentences). Thank you! $\endgroup$ – hoang tran Jan 26 at 8:38
  • 1
    $\begingroup$ as far as i know the grammar models can be adjusted to work without explicit punctuation present $\endgroup$ – Nikos M. Jan 26 at 8:45
  • $\begingroup$ Could you please name a few grammar models? It would be great if you could also include some links. Thank you very much! $\endgroup$ – hoang tran Jan 26 at 8:51
  • 1
    $\begingroup$ I dont have links to such models handy and it depnds on language spoken, but would be easy to search online or even construct one from scratch $\endgroup$ – Nikos M. Jan 26 at 8:53
  • 1
    $\begingroup$ a quick search gave me this; nlp.stanford.edu/software/lex-parser.shtml $\endgroup$ – Nikos M. Jan 26 at 8:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.