1
$\begingroup$

Documents, especially technical ones often contain non-text content in blocks (code snippets, os commands etc). Is there any efficient way to identify the starting and ending lines of those blocks?

| improve this question | | | | |
$\endgroup$
  • $\begingroup$ Have you seen LSTM models trained for recognizing punctuation marks? $\endgroup$ – Media Feb 11 '18 at 21:38
1
$\begingroup$

Although I didn't implement it so far, I am pretty sure natural language text vs code snippets is easy:

For each block, you make compare the distribution of characters to ground-truth natural language text vs. code. See my paper The WiLI benchmark dataset for written language identification, page 4 "Single-Character Frequency Analysis".

| improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.