Documents, especially technical ones often contain non-text content in blocks (code snippets, os commands etc). Is there any efficient way to identify the starting and ending lines of those blocks?
1 Answer
$\begingroup$
$\endgroup$
Although I didn't implement it so far, I am pretty sure natural language text vs code snippets is easy:
For each block, you make compare the distribution of characters to ground-truth natural language text vs. code. See my paper The WiLI benchmark dataset for written language identification, page 4 "Single-Character Frequency Analysis".