I have a list of some 100 millions of strings, each of different length. Examples: nsdgnlnesef ngmrlxkvgrmksefsfnlj rnrfnmsbanana housemgslen assistremovecouch

As you can see the strings vary in length and some can include english words, some not, some both words and random characters. All use the characters [a-z]. It is possible to characterise the strings are being humanly created or by a algorithm with a reasonable certainty.

Later I aim to also classify between different sub-types of algorithms generating these strings (multiclass classification).

The 100 millions strings I have are already labeled, based on other analysis. But because this is a very slow task, I will use Machine Learning to try and predict the origin of new strings as we get new data.

The methods I have started with are bigrams, Long Short Term Memory (LSTM) and Gated Recurrent Unit (GRU). I will eventually also run trigrams.

As both time and accuracy are very important factors of the method I eventually choose, I want to make sure I test as many as possible. But since there is a forest of methods, I hope to get some helpful advice here.

I imagine methods based on features, such as Random Forest and Support Vector Machine could be useful, but I haven't them used before, and I am not certain if they or other methods could handle 100 millions of strings in an efficient way.

Any suggestions for methods to look at are very welcomed.

  • $\begingroup$ How does a human make a distinction between human and a computer generated word? In your case is it the meaning? $\endgroup$ May 8, 2017 at 14:00
  • 1
    $\begingroup$ Some examples would be very nice, what determines if it's fake or not really points towards a group of solutions more so than that it is a set of strings $\endgroup$ May 8, 2017 at 14:02
  • $\begingroup$ The strings are often random characters, or a mix between that and dictionary words. It is possible to identify these strings as generated by human or algorithm using character distributions, and possibly also other features such as length of string etc. The strings were identified at a much later stage as generated by human or algorithm, the idea with using Machine Learning methods is to find quicker ways to make accurate predictions. $\endgroup$
    – Frank
    May 8, 2017 at 14:13
  • $\begingroup$ datascience.stackexchange.com/q/16115/8560 $\endgroup$
    – D.W.
    May 8, 2017 at 23:09


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.