1
$\begingroup$

I want to build a basic language detector for English, French and German.

I went to wikipedia and I downloaded the page of 'Technology' in all these languages.

In all these cases, we are talking for about 10000 words.

So basically I have 3 documents of 10000 words each for each of the 3 languages above.

My question is the following:

Should I split these documents in smaller documents e.g. of 100 words and create in this way more labeled observations in my dataset or should I leave them like this for training my classifier (e.g. with a TF-IDF model)?

Improve this question
$\endgroup$
2
  • $\begingroup$ Why don't you cluster them (as splitting) by their ends? Some sounds are language telling $\endgroup$
    – krayyem
    Commented May 31, 2019 at 0:59
  • $\begingroup$ @krayyem, ok but still this is not exactly the question of my post. $\endgroup$
    – Outcast
    Commented May 31, 2019 at 12:06

1 Answer 1

Reset to default
1
$\begingroup$

I think it should be easy to detect a language based on a standard vocabulary (bag of words).

However, I would split the articles into small pieces (maybe not 100 but a bit more words, eg. 200-500), so you can train and test your model without problems.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for your answer (upvote). My question principally why is it good to do something like that (not only what I have to do). My question is also related to this question (datascience.stackexchange.com/questions/52925/…) which I posted and it illuminates another aspect of this problem $\endgroup$
    – Outcast
    Commented May 31, 2019 at 9:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.