1
$\begingroup$

So I have a scraper that gets articles. However, it doesn't always work properly. I want to get better at checking when it doesn't work. For example, the following is something like I want it to scrape:

Hello. This is a sequence of sentences that are put together. They don't have to follow this exact format, but something very close to this would be nice! Just basically stuff like this put together with the occasional weird formatting, which depends on what is scraped.

But I might also get something that is obviously not text:

REGISTER | LOGIN | LOGOUT | Sign in to your account Forgot your password? {* #signInForm *}....

Is there any python library that checks the general format of strings? Basically, I am scraping articles and want to see if the text scraped is article-y. If there isn't a python library, would the best way to go be some sort of regex matching? Is this possible to do reasonably well?

Any help would be greatly appreciated, thanks!!

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ If your non-article scraps are not totally random, one fast way is to manually select some strings as flags (e.g. "#signInForm", ...) that probably denote a non-article and use them to discard the documents. This can be easily implemented and fast checked and will save you the trouble of doing something more sophisticated, such as outliers detection or document clustering.. So if possible, I would try the simpler approach first. $\endgroup$
    – Bogas
    Commented Jul 10, 2017 at 9:17
  • $\begingroup$ @Bogas Thanks for the suggestion! However, scraps are usually random to the point where it's very hard to guess... That being said, there might be a select few I can detect by doing this method. $\endgroup$
    – ocean800
    Commented Jul 10, 2017 at 14:40

1 Answer 1

Reset to default
1
$\begingroup$

I would try a semi-supervised learning technique where it passes you scraps and asks you to label them. What you're looking for will likely be kind of domain specific depending on the type of site. In the end you'll probably have a bunch of heuristics like:

  • If length < 50 and contains "LOGOUT", "REGISTER","SIGN IN","LOGIN"
  • If count of "|" > 1
  • If count of all upper case words > 1
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.