1
$\begingroup$

I am trying to find a dataset in which dataset targets words that are difficult. I understand there would be different levels of difficulty for each individual , but if we considered an average individual, I want to detect the difficult words that would be present in a sentence.

Example: Yes, may be today's Britains are not responsible for some of these reparations but the same speakers have pointed with pride to their foreign aid - you are not responsible for the people starving in Somalia but you give them aid surely the principle of reparation for what is the wrongs that have done cannot be denied.

IN the above sentence, the model should detect the word - reparations successfully.

$\endgroup$
3
  • $\begingroup$ Defining "an average individual" would be difficult, and your example illustrates this pretty well: a native English speaker wouldn't find the word "reparations" very difficult because they know related words "repair", "repairing" and the suffix "ation" is very common. French speakers would find it very easy, although they might get confused about its meaning in English. It would also be quite familiar to Spanish and Italian speakers. $\endgroup$
    – Erwan
    Jan 23 at 22:31
  • $\begingroup$ That's right @Erwan . But what if we display some of the words and ask the user whether know the meaning or not and based on that we can create a ranking through which we can showcase meanings based on that ranking $\endgroup$ Jan 27 at 18:05
  • $\begingroup$ You might want to look at readability measures, and possible implementations like this. Usually readability is applied to a text, not to a single word. $\endgroup$
    – Erwan
    Jan 27 at 23:24
1
$\begingroup$

On the basis that difficult words are difficult because they are not commonly used, I think something as simple as TF-IDF would work well.

$\endgroup$
1
  • $\begingroup$ Hi Nicholas, Thanks for the suggestion. I read about TF-IDF and although it is a viable solution but it may not solve its purpose when there are no difficult words . As in when we have got a simple essay that uses simple words that even a normal reader could understand. Is there any system where one can rank whether a word is difficult or not . $\endgroup$ Sep 1 '20 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.