3
$\begingroup$

In particular, I'm looking for something that can distinguish between complete sentences and sentence-like grammatical structures like clauses and phrases. It would also be very helpful if a library with this functionality was shared, but even a link to a paper would be appreciated.

For example- I would like "False" to be returned for strings like:

'Via The Daily Currant,'

“Waiting for the rain to stop”

'查看更多请返回网站主页。'

'* * *'

I would like "True" to be returned for things like:

"This is a complete sentence."

"I don't think there will ever be a Half Life 3, unfortunately."

Thank you!

$\endgroup$
  • $\begingroup$ I think a simple regular expression search would do the task. $\endgroup$ – Ankit Seth Aug 23 '18 at 6:36
  • $\begingroup$ Just taking 100 most used words in every language will allow you to detect the language. Most popular word in English is "the", in German it is "der", "die", "das". In Italian "non" and "e". In Russian "и" and "я". It is pretty straight forward with multiple sentences where statistics work $\endgroup$ – keiv.fly Aug 26 '18 at 18:04
  • 1
    $\begingroup$ Hi guys, thank you for your suggestions. I made some clarifying edits to make my request more clear. $\endgroup$ – rmac16 Aug 27 '18 at 19:19
1
$\begingroup$

To account for all your samples, first check if the text is English at all (solution as others hinted).

If yes, then there is a question what makes a 'complete English sentence'.

From your (two) samples, one can deduce that there should be a subject and a verb in finite form in relation with it. To raise attention to some 'edge cases':

"Go!"

"Done!"

To check subject-verb existence, you can try to parse - determine syntactic structure of the text but as this is still a tricky task, you can use part of speech tagging (actually morhological analysis plus disambiguation but PoS tagging made it's way as a name for the task). E.g. SpaCy does this https://spacy.io/usage/linguistic-features (It does dependency parsing too actually but error rate would be higher than using 'just' PoS tagging).

Then you can check in the output if there is a verb in a finite form - VBD, VBP in VBZ in SpaCy tag set (https://spacy.io/api/annotation#pos-tagging).

$\endgroup$
2
$\begingroup$

There is a library for python

https://pypi.org/project/langdetect/

This field is called "language detection" and it is quite active.

$\endgroup$
  • $\begingroup$ Hi, thank you for sharing this link. It is close to what I am looking for, but not exactly. I am new to asking questions on this site and rewrote my request to make it more specific. $\endgroup$ – rmac16 Aug 27 '18 at 19:24
0
$\begingroup$

As mentioned before, there are already libraries for that, however if you want to dive a bit deeper, I suggest you to have a look at something called Naive Bayes.

It is a simple algorithm that anyone can understand, if that gets your attention and you want something more sophisticated, then search for RNN and LSTM...

Have fun.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.