Is there a ready-made classifier to determine if webpage is an article or not?

Question

I am trying to solve a problem, where I have a set of URLs and I need to filter out only those that can be classified as an article (i.e. typically not include privacy policies, terms of usage, webpage for unsubscribing from a feed etc.).

Is there a trained classifier that can be used to filter these webpages? Every webpage classification material I found was about classifying articles into categories, but this doesn't concern me. I need to go a step back and figure out if the webpage is actually an article or not.

If there's not ready made classifier for this, what would be the best approach to create such classifier?

UPDATE: To clarify some questions below. The problem is not to identify an article only by the url, but to find a way of labelling a webpage as article/non-article based upon its contents (html, visual representation?, some other metadata perhaps). From the research I've done since asking the question, I believe this to be a variant of something called blog identification. I understand that the definition of an article is an important piece here and very ambiguous requirement, however that is exactly the hard part I have been struggling with while searching for some algorithm to do just that. If I had a fixed dataset, I should be able to identify common patterns and come up with a definition of an article by myself. But since the set of webpages that need to be filtered is very dynamic and unknown beforehand, I don't want to rely on my own custom rules for defining how an article looks like but would much rather use a method of identification that might have some research or at least a field-test validation behind it. It just seems to me, that this is a problem that someone had to be solving before me and that there might be solutions based upon that already.

Answer 1

There are some APIs that do that, like Diffbot's Analyze API.
As far as I'm aware there are no opensource readily-available models to do this, so if you're not going to use an API, you will have to create your own model.

Answer 2

I'm going on the assumption that you "have a set of URLs" that you need to pre-filter prior to doing something else with. Based on this assumption you could use urlparse and some string patterns to filter out X percentage of the URLs in your set that are most likely non-articles.

import re as regex
from urllib.parse import urlparse

urls =['https://candid.org/privacy-policy',
       'https://happify.com/health/privacy-policy',
       'https://www.abstract.com/legal/customer-terms-of-service',
       'https://pesa.org.au/membership/terms-of-service-and-privacy-statement',
       'https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html',
       'https://www.nasa.gov/content/nasa-rss-feeds',
       'https://www.sciencedaily.com/newsfeeds.htm',
       'https://www.ndtv.com/rss']

patterns = ['privacy-policy', 'customer-terms-of-service', 'terms-of-service-and-privacy-statement', 'rss-feeds',
            'newsfeeds', 'rss']
for url in urls:
    split_url = urlparse(url)
    possible_article = [pattern for pattern in patterns if regex.findall(pattern, split_url.path)]
    if not possible_article:
        print(f'Possible article: {url}')
        # output 
        Possible article: https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html

You could potentially expand the regular expression above to flag other URLs that meet common article URLs that can date strings or common keywords, such news or wirestory.

import re as regex
from urllib.parse import urlparse

urls =['https://www.ndtv.com/rss',
       'https://candid.org/privacy-policy',
       'https://www.sciencedaily.com/newsfeeds.htm',
       'https://happify.com/health/privacy-policy',
       'https://www.nasa.gov/content/nasa-rss-feeds',
       'https://www.bbc.com/news/technology-55675826',
       'https://www.abstract.com/legal/customer-terms-of-service',
       'https://pesa.org.au/membership/terms-of-service-and-privacy-statement',
       'https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html',
       'https://www.cnet.com/news/samsungs-galaxy-s21-upgrades-likely-wont-spell-an-end-to-galaxy-fe-or-note-lines-yet',
       'https://abcnews.go.com/Politics/wireStory/trump-leave-washington-morning-bidens-inauguration-75278801?cid=clicksource_4380645_6_heads_hero_live_headlines_hed']

non_article_patterns = ['privacy-policy', 'customer-terms-of-service', 'terms-of-service-and-privacy-statement',
                        'rss-feeds', 'newsfeeds', 'rss']

known_article_patterns = ['\d{4}\/\d{2}\/\d{2}', 'news', 'wireStory']

for url in urls:
    split_url = urlparse(url)
    non_article = [pattern for pattern in non_article_patterns if regex.findall(pattern, split_url.path)]
    if non_article:
        pass
    else:
        possible_article = [pattern for pattern in known_article_patterns if regex.findall(pattern, split_url.path)]
        if possible_article:
            print(f'Possible article: {url}')
            Possible article: https://www.bbc.com/news/technology-55675826
            Possible article: https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html
            Possible article: https://www.cnet.com/news/samsungs-galaxy-s21-upgrades-likely-wont-spell-an-end-to-galaxy-fe-or-note-lines-yet
            Possible article: https://abcnews.go.com/Politics/wireStory/trump-leave-washington-morning-bidens-inauguration-75278801?cid=clicksource_4380645_6_heads_hero_live_headlines_hed

BUT as Erwan pointing out in his comment it's not clear "how you define the difference between article and not article", so my answer is only part of a potential solution to your problem of URL classification.

It's hard to make any solid recommendation for any "out-of-the-box" solution that can meet an undefined use case.

Here are some links that might help you:

• paper: Fast webpage classification using URL features
• paper: A Comprehensive Study of Features and Algorithms for URL-Based Topic Classification
• paper: Web Page Classification: Features and Algorithms
• commercial: Website Categorization API

I'm still of the mindset that your use case will likely require a multi-prong approach and will need to be refined as more URLs are classified as either "an article or non-article."

Answer 3

There is not going to be a pretrained classifier for article/not article because different people will have different definitions of article/not article.

The first step is define article/not article for your specific use case. That can be done two ways:

1. Series of rules - List criteria for article/not article, Then encode those rules in a program. For example an article does not contain the phrase "Terms of service". This strategy works well for deterministic, narrow definitions that do not change.

2. Train a binary classifier - Label thousands of webpages as article/not article. Then preprocess them so they are amenable for machine learning. Then fit a binary classifier, Naive Bayes classifier is a common first choice. This strategy works well for probabilistic, complex definitions that can change.