1
$\begingroup$

Documents, especially technical ones often contain non-text content in blocks (code snippets, os commands etc). Is there any efficient way to identify the starting and ending lines of those blocks?

$\endgroup$
1
  • $\begingroup$ Have you seen LSTM models trained for recognizing punctuation marks? $\endgroup$ Feb 11, 2018 at 21:38

1 Answer 1

1
$\begingroup$

Although I didn't implement it so far, I am pretty sure natural language text vs code snippets is easy:

For each block, you make compare the distribution of characters to ground-truth natural language text vs. code. See my paper The WiLI benchmark dataset for written language identification, page 4 "Single-Character Frequency Analysis".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.