0
$\begingroup$

I have strings short texts of 2 types: '23jd2032n0d2mn', 'fn830n30rn83', 'fhui29n4ok', 'qn4foml', ... and 'sweetie23', 'king3prussia', 'maryjesus', 'lovedog4and_kitties', ...

Is there a way to distinguish one type from another?

I've tried to vectorize texts with word2vec and classify on these vectors with xgboost, but I didn't succeded to achieve got F1-score.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
2
$\begingroup$

You could train a character-level language model, e.g. an LSTM, on the real short texts, and use the perplexity as the signal to know whether a piece of text is real or not.

In order to find an appropriate perplexity threshold, you can have a look at the distribution of perplexities over a validation holdout dataset.

UPDATE: There are multiple implementations of language models. For "classical" options, you can go for KenLM or if you have GPUs to train the model, you can use fairseq. Just remember to prepare your text to have character-level tokens (normally you just need to have a space between every letter).

Improve this answer
$\endgroup$
3
  • $\begingroup$ I'd defintely try this approach but first I need to learn it better. If u have any links to read on this topic, could u share with me, please $\endgroup$
    – Marcel Mars
    Sep 22, 2020 at 16:53
  • $\begingroup$ I added an update to the answer with a couple of suggestions about language model implementations. $\endgroup$
    – noe
    Sep 22, 2020 at 17:00
  • $\begingroup$ Also, if you found the answer useful, please consider marking is as correct. $\endgroup$
    – noe
    Sep 22, 2020 at 17:00
1
$\begingroup$

Assuming that the "human readable" texts are more likely to contain actual words, you could count the number of dictionary words that occur in each.

You could use Wordnet for example.

The number or proportion of word hits, and their length, could be features for a model or maybe it would be enough with a simple cutoff rule.

You might want to restrict the word list to the most frequent words in the language you're working with.

Improve this answer
$\endgroup$
1
  • $\begingroup$ that might work, but 2nd type of texts may contain human readable words with mistakes, numbers, or words without wovels, or names with special characters without space, etc $\endgroup$
    – Marcel Mars
    Sep 22, 2020 at 16:51
0
$\begingroup$

Try making features like vowel_count, consonant_count, digitcount , vowel_density(vowel_count/total_length_of_words)

Another wild thing - 
split the strigns with numbers and _ using regex and try to see if they are english words or not, use a pretrained model like spacy.english or nltk.words to check, make a column representing english words count if any.

edit -

also find the vowel index from last vovel index(relative index) (after removing nonnumeric characters)and eg - 
    A) maryjesus -> a(1, because 1st index), e(5th index - index of a(1st index) = 4), u(2)
    
    B)qn4foml-> o(4)

because in english words , vowels ties the consonant together to form words, therefore an english words vowel's relative distance should be quite less.

Improve this answer
$\endgroup$
5
  • $\begingroup$ This looks like an interesting project, what is this for ? I'd like to collaborate if you need help. $\endgroup$
    – Madhur Yadav
    Sep 16, 2020 at 19:05
  • $\begingroup$ You can also add an IS_NER column and check if the words after splitting are NERs or not. $\endgroup$
    – Madhur Yadav
    Sep 16, 2020 at 19:10
  • 1
    $\begingroup$ thanks for ideas, Madhur! I'll try and tell you the results $\endgroup$
    – Marcel Mars
    Sep 16, 2020 at 19:21
  • 1
    $\begingroup$ Character based heuristics helped. I succeded to reach 0.8 F1-score on cross validation. I've used both vectors and manually created features. But I will try LSTM-approach as well. $\endgroup$
    – Marcel Mars
    Sep 18, 2020 at 10:18
  • 1
    $\begingroup$ great, keep us posted, also if its a personal project then do share it in github $\endgroup$
    – Madhur Yadav
    Sep 18, 2020 at 10:21

Not the answer you're looking for? Browse other questions tagged or ask your own question.