I want to build a basic language detector for English, French and German.

I went to wikipedia and I downloaded the page of 'Technology' in all these languages.

In all these cases, we are talking for about 10000 words.

So basically I have 3 documents of 10000 words each for each of the 3 languages above.

My question is the following:

Should I split these documents in smaller documents e.g. of 100 words and create in this way more labeled observations in my dataset or should I leave them like this for training my classifier (e.g. with a TF-IDF model)?

  • $\begingroup$ Why don't you cluster them (as splitting) by their ends? Some sounds are language telling $\endgroup$ – krayyem May 31 '19 at 0:59
  • $\begingroup$ @krayyem, ok but still this is not exactly the question of my post. $\endgroup$ – Outcast May 31 '19 at 12:06

I think it should be easy to detect a language based on a standard vocabulary (bag of words).

However, I would split the articles into small pieces (maybe not 100 but a bit more words, eg. 200-500), so you can train and test your model without problems.

| improve this answer | |
  • $\begingroup$ Thank you for your answer (upvote). My question principally why is it good to do something like that (not only what I have to do). My question is also related to this question (datascience.stackexchange.com/questions/52925/…) which I posted and it illuminates another aspect of this problem $\endgroup$ – Outcast May 31 '19 at 9:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.