3
$\begingroup$

In particular, I'm looking for something that can distinguish between complete sentences and sentence-like grammatical structures like clauses and phrases. It would also be very helpful if a library with this functionality was shared, but even a link to a paper would be appreciated.

For example- I would like "False" to be returned for strings like:

'Via The Daily Currant,'

“Waiting for the rain to stop”

'查看更多请返回网站主页。'

'* * *'

I would like "True" to be returned for things like:

"This is a complete sentence."

"I don't think there will ever be a Half Life 3, unfortunately."

Thank you!

Improve this question
$\endgroup$
3
  • $\begingroup$ I think a simple regular expression search would do the task. $\endgroup$
    – Ankit Seth
    Commented Aug 23, 2018 at 6:36
  • $\begingroup$ Just taking 100 most used words in every language will allow you to detect the language. Most popular word in English is "the", in German it is "der", "die", "das". In Italian "non" and "e". In Russian "и" and "я". It is pretty straight forward with multiple sentences where statistics work $\endgroup$
    – keiv.fly
    Commented Aug 26, 2018 at 18:04
  • 1
    $\begingroup$ Hi guys, thank you for your suggestions. I made some clarifying edits to make my request more clear. $\endgroup$
    – shoshi
    Commented Aug 27, 2018 at 19:19

3 Answers 3

Reset to default
1
$\begingroup$

To account for all your samples, first check if the text is English at all (solution as others hinted).

If yes, then there is a question what makes a 'complete English sentence'.

From your (two) samples, one can deduce that there should be a subject and a verb in finite form in relation with it. To raise attention to some 'edge cases':

"Go!"

"Done!"

To check subject-verb existence, you can try to parse - determine syntactic structure of the text but as this is still a tricky task, you can use part of speech tagging (actually morhological analysis plus disambiguation but PoS tagging made it's way as a name for the task). E.g. SpaCy does this https://spacy.io/usage/linguistic-features (It does dependency parsing too actually but error rate would be higher than using 'just' PoS tagging).

Then you can check in the output if there is a verb in a finite form - VBD, VBP in VBZ in SpaCy tag set (https://spacy.io/api/annotation#pos-tagging).

Improve this answer
$\endgroup$
2
$\begingroup$

There is a library for python

https://pypi.org/project/langdetect/

This field is called "language detection" and it is quite active.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi, thank you for sharing this link. It is close to what I am looking for, but not exactly. I am new to asking questions on this site and rewrote my request to make it more specific. $\endgroup$
    – shoshi
    Commented Aug 27, 2018 at 19:24
0
$\begingroup$

As mentioned before, there are already libraries for that, however if you want to dive a bit deeper, I suggest you to have a look at something called Naive Bayes.

It is a simple algorithm that anyone can understand, if that gets your attention and you want something more sophisticated, then search for RNN and LSTM...

Have fun.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.