1
$\begingroup$

I am trying to find a dataset in which dataset targets words that are difficult. I understand there would be different levels of difficulty for each individual , but if we considered an average individual, I want to detect the difficult words that would be present in a sentence.

Example: Yes, may be today's Britains are not responsible for some of these reparations but the same speakers have pointed with pride to their foreign aid - you are not responsible for the people starving in Somalia but you give them aid surely the principle of reparation for what is the wrongs that have done cannot be denied.

IN the above sentence, the model should detect the word - reparations successfully.

Improve this question
$\endgroup$
3
  • $\begingroup$ Defining "an average individual" would be difficult, and your example illustrates this pretty well: a native English speaker wouldn't find the word "reparations" very difficult because they know related words "repair", "repairing" and the suffix "ation" is very common. French speakers would find it very easy, although they might get confused about its meaning in English. It would also be quite familiar to Spanish and Italian speakers. $\endgroup$
    – Erwan
    Commented Jan 23, 2021 at 22:31
  • $\begingroup$ That's right @Erwan . But what if we display some of the words and ask the user whether know the meaning or not and based on that we can create a ranking through which we can showcase meanings based on that ranking $\endgroup$
    – Abir Pattnaik
    Commented Jan 27, 2021 at 18:05
  • $\begingroup$ You might want to look at readability measures, and possible implementations like this. Usually readability is applied to a text, not to a single word. $\endgroup$
    – Erwan
    Commented Jan 27, 2021 at 23:24

1 Answer 1

Reset to default
1
$\begingroup$

On the basis that difficult words are difficult because they are not commonly used, I think something as simple as TF-IDF would work well.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi Nicholas, Thanks for the suggestion. I read about TF-IDF and although it is a viable solution but it may not solve its purpose when there are no difficult words . As in when we have got a simple essay that uses simple words that even a normal reader could understand. Is there any system where one can rank whether a word is difficult or not . $\endgroup$
    – Abir Pattnaik
    Commented Sep 1, 2020 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.