2
$\begingroup$

I'm attempting a rather simple exercise in machine learning and trying to classify samples of text as either plaintext or ciphertext (encrypted).

Here are two samples:

  • Plaintext: This is a sentence in plaintext which any human person can read
  • Ciphertext: 5oXbLiEZbMUgOOdYy+q4+rsDaqUngBrrUbpVeuu2ggvP6hHObC4GgTLhq

The specific encryption used doesn't have any special attributes I can use to classify (for example, the ciphertext isn't guaranteed to be significantly longer than plaintext), so the task is all about figuring out what text is indistinguishable from random characters and what text is readable plaintext.

My current heuristic involves counting whitespaces and assuming anything with a whitespace ratio above a certain threshold is plaintext, but I'm trying to find a better robust algorithm.

$\endgroup$
1
  • $\begingroup$ What forms of plaintext are you needing to test? I.e. what is the full scope of inputs that your algorithm hopes to classify? Some things will be next to impossible (such as detecting whether a compressed file is also encrypted), but the simple English example you give is very easy. $\endgroup$ – Neil Slater May 5 '17 at 13:00
2
$\begingroup$

Simply counting the frequency of characters ought to easily distinguish between English language and ciphertext, because they're so obviously different.

You can just count the frequency of characters in a big corpus of English, and a big corpus of ciphertext, and apply a chi-squared test to each to figure out which one matches the counts in a new chunk of text.

Or if you can assume ciphertext has a roughly uniform distribution over characters, that alone lets you construct a good test for whether new text is unlikely to be ciphertext.

I did a short blog post on something similar. https://blog.cloudera.com/blog/2016/09/solving-real-life-mysteries-with-big-data-and-apache-spark/

$\endgroup$
1
  • 1
    $\begingroup$ This is a form of frequency analysis, most sophisticated encryption routines will defeat this through padding. If it is binary result (English|Encrypted) then this is a language detection problem, which is fairly well solved in any programming language. Also consider asking this question at crypto.stackexchange.com . $\endgroup$ – davmor Nov 20 '18 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.