2
$\begingroup$

My native language is a regional language and few people speak it. I have some assignements in a machine learning course and i was thinking about doing some natural languge processing on my native language but i don't know where to start since there is almost no research about this language ( no corpus , no research papers , ... ) and i'm new to machine learning.

I want to start doing everything from bottom and i want to do things right , can you please guide me to steps i should follow?

I also want to build my own corpus ,since it's a very tiring work , is there a way to build a single corpus that can be used on several NLP applications ( at least for translation and speach recognition) ?

Improve this question
$\endgroup$
1
  • $\begingroup$ What language are we talking about? $\endgroup$
    – Piotr Rarus
    Dec 4, 2019 at 14:41

2 Answers 2

Reset to default
1
$\begingroup$

The key term for your problem is low resource languages. I'm not sure whether there is a standard approach but you could find papers about what people have done before in similar cases, potential software tools/data repositories which could help, etc.

You might also be interested in the Universal Dependencies project: https://universaldependencies.org/

There are many things that can be done related to Universal Dependencies (UD):

  • use the existing resources to analyze/parse some text data. As far as I know the standard tool would be UDPipe (python libraries here, here, maybe others...)
  • train a new dependency parser. for instance I found this repository.
  • start a treebank for a new language: https://universaldependencies.org/how_to_start.html. Warning: this is probably a lot of work if you start from scratch! and there's no or little ML involved in the creation of the data itself.

The focus of UD resources is on dependencies but imho the main interest is that it provides resources which can be used for all the standard NLP tasks: sentence or word segmentation, POS tagging and lemmatization, etc.

Improve this answer
$\endgroup$
2
  • $\begingroup$ How the universaldependencies project can be used with Python and some NLP techniques? I tried to look at their site, but it doesn't seem there is a kind of dictionary ready to be used. I'd assume, but not tested yet, I need to download a file and get it digested into a dictionary/vector with whichever tool I want to use (e.g. NLTK). $\endgroup$
    – Andrea Moro
    Dec 9, 2019 at 8:04
  • $\begingroup$ @AndreaMoro I added some links in my answer, hope this helps $\endgroup$
    – Erwan
    Dec 9, 2019 at 9:43
0
$\begingroup$

@Amar, you can start with NLTK, Spacy, Flair and Polyglot to start with your NLP learning. They are easier to start and understand. You can create your own model for your language and contribute if you have the corpus available with you. The nouns, verbs are easier to capture in English when compared to Chinese, Korean and Arabic. Every language has it own beautiful representation of context. You would a good library that can read the language that you are developing. You can also try IBM Watson and AWS SageMaker on trial basis with check for Language detection.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.