1
$\begingroup$

I am trying to compile a Lexical Simplification dataset, which contains sentences, target words, and their simpler substitutes. I have already found some similar datasets in various languages, here are some of them for reference:

  • English - contains 1 target word per sentence
  • Spanish - contains multiple target words per sentence
  • Spanish - contains multiple target words per sentence
  • English - contains 1 target word per sentence

However, none of these sources seem to state how the target word was selected. Target words are words that are potentially complex, meaning that they may be labelled as complex or not complex. Are they chosen arbitrarily? How would such a process typically be carried out?

Thanks in advance!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Word frequency is the best way to identify complex words.

You have also other factors like word length, number of syllables, number of synonyms, or the complexity of pronunciation.

Then, those words are processed in Decision Trees to find the best solution for simplification.

See also: https://aclanthology.org/S16-1085.pdf

About the data source, they are in the test/train sample datasets.

https://github.com/MMU-TDMLab/CompLex/blob/master/test/lcp_multi_test.tsv

enter image description here

For instance, europarl refers to the european parliament datasets:

https://www.statmt.org/europarl/

Improve this answer
$\endgroup$
4
  • $\begingroup$ Hi Nicolas, thank you very much for taking the time to answer my question. Yes, I have read that these are features typically used for the CWI task. However, my question relates to the compilation of a dataset, which I guess should be made up of both complex words and non complex words. Apologies, perhaps I should have phrased it better in the initially question. As in the datasets I have linked, the new dataset should include words not deemed as complex, so that the system being implemented may distinguish between complex and non complex words and thus not generate substitutes for them. $\endgroup$
    – Mar
    Commented Dec 15, 2022 at 10:06
  • $\begingroup$ Do you mean the data sources? Then, there are algorithms that classify every word automatically: Do you mean those algorithms? $\endgroup$
    – Nicolas Martin
    Commented Dec 15, 2022 at 21:35
  • $\begingroup$ No I mean the data sources. $\endgroup$
    – Mar
    Commented Dec 16, 2022 at 10:44
  • $\begingroup$ OK Mar, I've updated the answer accordingly. $\endgroup$
    – Nicolas Martin
    Commented Dec 16, 2022 at 12:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.