Removing junk sentences

Question

I have transcripts of phone calls with customers and agents. I'm trying to find promises which were made by an agent to a customer.

I already did punctuation restoration. But there are a lot of sentences that don't have any sense. I would like to remove them from the transcript. Most of them are just a set of not connected words. I wonder what approach is the best for this task?

My ideas are:

• Use tf idf and word2vec to create vectors from all sentences. After that we can do some kind of anomaly detection e.g. look for and delete vectors that are highly deviated from most other vectors.

• Spam filters. Maybe is it possible to apply spam filters for this task?

• Crate some pattern of part of speech tags that proper sentence must include. For example, any good sentence must include noun + verb. Or we can use for example dependency tokens from spacy.

update

Example of a sentence that I want to keep:

There's no charge once sent that you'll get a ups tracking number.

Example of a junk sentence:

Kinder pr just have to type it in again, clock drives bethel.

Another junk sentence:

Just so you have it on and said this is regarding that.

Answer 1

I would suggest training your own custom Dialog Act Classification Model. Detecting a dialog act is a Natural Language Understanding (NLU) problem. Using a dialog act classification model you can detect if a sentence is used for "greeting", "question", "opinion", "promise", etc.

Detailed approach:

Use a pre-trained language model (for example BERT), to train your own Dialog act classification model. You can create your own custom Dialog act class based on your requirement (as you want to remove specific types of sentences). You can class them as junk or use "greeting", "question", "opinion", etc as dialog class.

After you are done with the training you can loop through the list of sentences and filter them based on their predicted classes.

You might have to research if there are similar corpus which you can use or have to manually label and then train the model. It will be a more reliable approach as compared to the word2vec or rule-based POS approach.

For more details on state-of-the-art approaches and dialogue act corpus follow this link. If you have no idea how to use a language model you can use the transformers library from huggingface. I hope you find this helpful.

Answer 2

I am answering to the question in your title ("removing junk sentences"), ignoring your final goal of finding promises within the corpus.

One thing I would try is to treat this as a classification problem (junk vs non-junk). You can train a model based on a labelled set (i.e. you need to label some subset of your dataset) and then classify the rest of the corpus. You could use a pre-trained language model like Bert and fine-tune it with you labeled set, as in here (https://colab.research.google.com/github/google-research/bert/blob/master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb).

The advantage of using a language model like this is that you don't have to worry too much about linguistic (pre-)processing, meaning you don't have to get the part-of-speech or syntactic structure.

Comments regarding your ideas:

1. Anomaly detection with tf-idf and word2vec: It depends on the proportion of the junk sentences in your corpus. If they it's more than 15%, I would think that they might not be so anomal. Also, I am assuming your junk sentences come from noisy automatic speech-to-text transcription. I am not sure, to what extent parts of these junk sentences are correctly transcribed and what the effect of the correctly transcribed portion might have on the extent of the anomaly.

2. If you mean pre-existing spam filters that are trained on spam email, I would guess that the spammyness of emails is quite different from junkiness of your transcripts.

3. Use POS tags or syntactic structure to manually create rules for valid sentences: This seems a bit tedious too me and also I am not sure if you will discover all junk with this. For instance, in your junk examples, the syntactic structure does not strike me as too unusal, e.g. "clock drives bethel" might be tagged as , which is quite a common tag sequence. The junkiness in this case comes from the meaning of the words.