2
$\begingroup$

Can some one advise me direction where to look in.Or some resources. Here is a task:

  1. User leaves feed back-text with min 50 characters.
  2. I need to check if it's normal human sentences/ word combination OR just bag of words and characters.

For ex ( 1-normal, 0-not normal):

"I wrote question.hope for answer" - 1(class)

"Bla bla goog goog goog gooo" - 0(class)

Maybe some dataset available.or some approach? Thanks in advance!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

What you need is simply a language model. This is a very common task so you should be able to find code and data easily. This question gives some pointers for Python (be careful, the accepted answer is incorrect according to the two other answers).

Applying the language model to a sentence gives you a probability (or a perplexity score, which works the opposite way), so you have to define a threshold in order to classify as real language or not.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Let me clarify bit: my task is simple classification task, where I use LM and dataset with (readable/not readable) samples? Same as if I would wanted to classify for Pos/Neg sentiment? In such case I can use fast.ai or bert+keras. $\endgroup$
    – Serhiy
    Jun 20, 2020 at 13:41
  • 1
    $\begingroup$ Yes but if you want to properly check if the text is readable you need to use a model which uses the words as features, that's what a language model does. The easiest way is probably to use a pre-trained model as described in the linked question. If you want an even simpler option you can just check whether the words appear in a dictionary, but that's quite limited. $\endgroup$
    – Erwan
    Jun 20, 2020 at 13:57
  • $\begingroup$ Or I can use gpt as the last comment in question you sent me @Erwan. tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt') $\endgroup$
    – Serhiy
    Jun 20, 2020 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.