In NLP, languages are often referred as low resource
or high resource
.
What these terms mean ?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this communityIn NLP, languages are often referred as low resource
or high resource
.
What these terms mean ?
High resource languages are languages for which many data resources exist, making possible the development of machine-learning based systems for these languages. English is by far the most well resourced language. West-Europe languages are quite well covered, as well as Japanese and Chinese. Naturally low-resource languages are the opposite, that is languages with none or very few resources available. Unsurprisingly many languages from Africa and other poorest parts of the world are low-resources. But this is also the case for some extinct or near-extinct languages and many local dialects. There are actually many languages which are mostly oral, for which very few written resources exist (let alone resources in electronic format); for some there are written documents but not even something as basic as a dictionary.
There are many different types of resources which are needed in order to train good language-based systems:
Many types of language resources are costly to produce, this is why the economic inequalities between countries/languages are reflected in the amount (or absence) of language resources. The Universal Dependencies project is an interesting effort to fill this gap.