Determining whether a sentence is "cliche" using NLP

Question

I have a collection of essays from students. Each essay is about the same topic and of the same word length. My goal is to develop a machine learning algorithm that pinpoints "cliche" sentences (i.e., sentences that are "unoriginal" and similar to what other students have written).

Let document X be the essay we're trying to analyze. Let S be the set of all essays. Here is the rudimentary approach I've thought of:

1. Split all essays (from set S - X) into individual sentences. Call this collection of sentences Y. Use this solution to compare each sentence of X to each sentence in Y. This will yield an array of scores Z for each sentence in X.
2. For each Z: weight each score in Z based on position. If a sentence in X and a sentence in Y are in similar locations in their respective essays (e.g., both at the intro of the essay) the weight on the respective score in Z will be higher than if these two sentences were farther apart.
3. Average all of the Z arrays for all of the sentences. The sentence with the highest average is the "most cliche."

Is there a better approach to this problem?

Answer 1

IMO. Firstly, you make the hypothesis that cliche sentences are in the same part of the document, which makes sense. But how do you determine the weight of the position and what if the students use different structures (e.g., some write the conclusion and the discussion sections separate, others write them together)?

Then you could get more up to date sentence embeddings algorithms than the one you suggested to provide you with more accurate similarity scores.

Finally, how I would approach it. I would use a clustering technique such as DBScan or HDBScan (with a cosine distance). The reason I would use these is that they are distance-based clustering techniques. Hence, initially, you can start with a small aperture, distance, and view which sentences are very similar to each other, almost identical. Then you can increase your distance, open your aperture, and view which others are quite similar to them. You could of course try other clustering techniques and see how they would work since those two are usually not the best. Then I would take into consideration where each sentence position is and see if it would change the results, like that, I believe, you have more of an insight into what is happening.

Answer 2

You'll define a distance metric between phrases to find similar phrases between students.

Word Mover’s Distance (WMD) is an algorithm for finding the distance between phrases in a word embedding space.