2
$\begingroup$

I'm working on a large corpus of french daily newspapers from the 19th century that have been digitized and where the data are in the form of raw OCR text files (one text file per day). In terms of size, one year of issues is around 350 000 words long.

What I'm trying to achieve is to detect the different articles that form a newspaper issue. Knowing that an article can be two or thee lines long or very much longer, that there is no systematic typographic division and that there are a lot of OCR errors in each file. I should also mention that I don't have access to others OCR data like document layout in XML or so.

I've tried the TexTiling algorithm (the nltk implementation) but the results were not really conclusive.

Before diving deeper by myself I was wondering if maybe some of you would have some hint about a task like this one : train a machine learning model, try others algorithms ?

Improve this question
$\endgroup$
1
  • $\begingroup$ I would think that doing ML on the images to split them first and doing OCR afterwards is the only realistic way to do this automatically. $\endgroup$
    – Valentas
    Jan 20, 2021 at 21:30

1 Answer 1

Reset to default
0
$\begingroup$

As far as I know topic segmentation is not a particularly easy task with clean data, so it's likely to be challenging with noisy old French.

It's not exactly the same problem so I'm not sure if this is useful but you might want to look into using stylistic features in order to help the model detect the changes between articles. There has been a fair amount of work on the task of style change detection as part of the PAN series (the task has run for 3 years, results and papers are available from the previous years).

Hope this helps.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the PAN series reference, will look into it. Regarding the noisiness of the old french, it's not so much the language itself than the OCR errors which are problematics (19th century french is very close from present speaking habits) $\endgroup$
    – Tetro
    Aug 6, 2019 at 21:01
  • $\begingroup$ Sure I didn't mean old French like it's a different language, still it could be a problem for using a model trained on a modern French resource because there could be significant differences at least in the vocabulary distribution. $\endgroup$
    – Erwan
    Aug 6, 2019 at 22:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.