0
$\begingroup$

I am relatively new to data science and have an exercise task. This consists of the classifications of excerpts of texts. However, the texts are obfuscated such that one cannot read the words, spaces etc. But the "patterns" are preserved. I have a training set of the following form. One .txt file with the text excerpts looking like

shdbcjhbjhbefbhbwhbkbehbwbwbfhwb / wbhbwtjnwkjbrfbqlenk / wjnfkjebrkbrghkbibgibib / tberbtewtwbkwtjbrkbwkbtwrbt / . . .

and one .txt file with the labels of the texts looking like

1 / 4 / 11 / 0 / . . .

This means that each string of symbols belongs to a certain text and one knows to which one.

The task is now to set up a classifier, to train it on above training data and to test it on test data which also consists of text excerpts.

My basic idea was to interpret the data as images and to set up a deep learning NN with e.g. TensorFlow. However, I am not sure about the shape of the data which is needed to feed the data to the NN.

Do I have to create label-folders containing all texts with a certain class-label or is there a more direct way (so far I created vectors in R consisting of the texts and the labels)? Must the excerpts all have the same length? How do I have to manipulate the data until I can feed it to the NN?

Improve this question
$\endgroup$
  • $\begingroup$ I came across a similar problem and I am trying to encode the whole sequence in one hot encoding using character frequencies instead of words. What approach worked for you? I will post my findings here soon. $\endgroup$ – Anon Dec 21 '17 at 15:24
1
$\begingroup$

The text is obfuscated for you but to the computer, this is as incomprehensible as English for instance. When classifying text, you always first vectorize your input, meaning that your text becomes a vector. This vector typically represents the frequency of the tokens in your vocabulary in that particular string (bag-of-words representation).

What you can do is represent each document/string as a vector representing the frequencies of characters, character n-grams, or even words. One you have your input, you can deal with it as any classification task.

Naive Bayes has shown to be a good lower bound for these types of tasks.

Improve this answer
$\endgroup$
0
$\begingroup$

I came across a similar problem. So I thought instead of finding a solution to convert the text into something understandable, we can process directly with character-based model. This link will help you a lot https://offbit.github.io/how-to-read/

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.