I have a collection of essays from students. Each essay is about the same topic and of the same word length. My goal is to develop a machine learning algorithm that pinpoints "cliche" sentences (i.e., sentences that are "unoriginal" and similar to what other students have written).
Let document X be the essay we're trying to analyze. Let S be the set of all essays. Here is the rudimentary approach I've thought of:
- Split all essays (from set S - X) into individual sentences. Call this collection of sentences Y. Use this solution to compare each sentence of X to each sentence in Y. This will yield an array of scores Z for each sentence in X.
- For each Z: weight each score in Z based on position. If a sentence in X and a sentence in Y are in similar locations in their respective essays (e.g., both at the intro of the essay) the weight on the respective score in Z will be higher than if these two sentences were farther apart.
- Average all of the Z arrays for all of the sentences. The sentence with the highest average is the "most cliche."
Is there a better approach to this problem?