1
$\begingroup$

I am trying to solve a problem, where I have a set of URLs and I need to filter out only those that can be classified as an article (i.e. typically not include privacy policies, terms of usage, webpage for unsubscribing from a feed etc.).

Is there a trained classifier that can be used to filter these webpages? Every webpage classification material I found was about classifying articles into categories, but this doesn't concern me. I need to go a step back and figure out if the webpage is actually an article or not.

If there's not ready made classifier for this, what would be the best approach to create such classifier?

UPDATE: To clarify some questions below. The problem is not to identify an article only by the url, but to find a way of labelling a webpage as article/non-article based upon its contents (html, visual representation?, some other metadata perhaps). From the research I've done since asking the question, I believe this to be a variant of something called blog identification. I understand that the definition of an article is an important piece here and very ambiguous requirement, however that is exactly the hard part I have been struggling with while searching for some algorithm to do just that. If I had a fixed dataset, I should be able to identify common patterns and come up with a definition of an article by myself. But since the set of webpages that need to be filtered is very dynamic and unknown beforehand, I don't want to rely on my own custom rules for defining how an article looks like but would much rather use a method of identification that might have some research or at least a field-test validation behind it. It just seems to me, that this is a problem that someone had to be solving before me and that there might be solutions based upon that already.

$\endgroup$
7
  • 1
    $\begingroup$ So are you trying to do pre-filtering on the URLs before requesting content from them using requests.get? $\endgroup$ Jan 14 at 13:58
  • 1
    $\begingroup$ Welcome to DataScienceSE. I don't see clearly how you define the difference between "article" and "not article". For example, almost every website has a privacy policy, and for legal reasons they have to disclose it on their front page. Many news websites also have subscribe/unsubscribe links, even though their content is made of articles. $\endgroup$
    – Erwan
    Jan 14 at 14:11
  • 1
    $\begingroup$ I noted that the update part of your question is commonly asked on Stack Overflow, but most of the answers related to those question talk about the complexity of doing something "out-of-the-box." So far, I haven't discovered a python module that classifies the content of any webpage as article vs non-article. most classify a page based on topic type (e.g., food, car, etc.). $\endgroup$ Jan 15 at 22:53
  • $\begingroup$ It's unlikely that somebody has already solved a problem that cannot be defined clearly, and even if you found an implementation it would be hard to know if the solution actually solves the same problem as yours. The definition of "article" is quite vague, and there are major difference between scientific articles, news articles, lifestyle articles,... I don't think you can avoid defining what is meant by "article". Btw imho the visual representation and/or metadata is not very relevant. $\endgroup$
    – Erwan
    Jan 16 at 16:12
  • 1
    $\begingroup$ @koperko "Quasai's answer proved that the solution is already out there" Be careful, it's not because the description contains the word "article" that it solves the same problem as yours. Maybe you actually have a precise idea of what you need in mind, it's just that you don't find a way to express it. But if you don't it's risky to build a system which doesn't have a clear goal. $\endgroup$
    – Erwan
    Jan 20 at 17:16
1
+100
$\begingroup$

There are some APIs that do that, like Diffbot's Analyze API.
As far as I'm aware there are no opensource readily-available models to do this, so if you're not going to use an API, you will have to create your own model.

$\endgroup$
3
$\begingroup$

I'm going on the assumption that you "have a set of URLs" that you need to pre-filter prior to doing something else with. Based on this assumption you could use urlparse and some string patterns to filter out X percentage of the URLs in your set that are most likely non-articles.

import re as regex
from urllib.parse import urlparse

urls =['https://candid.org/privacy-policy',
       'https://happify.com/health/privacy-policy',
       'https://www.abstract.com/legal/customer-terms-of-service',
       'https://pesa.org.au/membership/terms-of-service-and-privacy-statement',
       'https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html',
       'https://www.nasa.gov/content/nasa-rss-feeds',
       'https://www.sciencedaily.com/newsfeeds.htm',
       'https://www.ndtv.com/rss']

patterns = ['privacy-policy', 'customer-terms-of-service', 'terms-of-service-and-privacy-statement', 'rss-feeds',
            'newsfeeds', 'rss']
for url in urls:
    split_url = urlparse(url)
    possible_article = [pattern for pattern in patterns if regex.findall(pattern, split_url.path)]
    if not possible_article:
        print(f'Possible article: {url}')
        # output 
        Possible article: https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html

You could potentially expand the regular expression above to flag other URLs that meet common article URLs that can date strings or common keywords, such news or wirestory.

import re as regex
from urllib.parse import urlparse

urls =['https://www.ndtv.com/rss',
       'https://candid.org/privacy-policy',
       'https://www.sciencedaily.com/newsfeeds.htm',
       'https://happify.com/health/privacy-policy',
       'https://www.nasa.gov/content/nasa-rss-feeds',
       'https://www.bbc.com/news/technology-55675826',
       'https://www.abstract.com/legal/customer-terms-of-service',
       'https://pesa.org.au/membership/terms-of-service-and-privacy-statement',
       'https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html',
       'https://www.cnet.com/news/samsungs-galaxy-s21-upgrades-likely-wont-spell-an-end-to-galaxy-fe-or-note-lines-yet',
       'https://abcnews.go.com/Politics/wireStory/trump-leave-washington-morning-bidens-inauguration-75278801?cid=clicksource_4380645_6_heads_hero_live_headlines_hed']

non_article_patterns = ['privacy-policy', 'customer-terms-of-service', 'terms-of-service-and-privacy-statement',
                        'rss-feeds', 'newsfeeds', 'rss']

known_article_patterns = ['\d{4}\/\d{2}\/\d{2}', 'news', 'wireStory']

for url in urls:
    split_url = urlparse(url)
    non_article = [pattern for pattern in non_article_patterns if regex.findall(pattern, split_url.path)]
    if non_article:
        pass
    else:
        possible_article = [pattern for pattern in known_article_patterns if regex.findall(pattern, split_url.path)]
        if possible_article:
            print(f'Possible article: {url}')
            Possible article: https://www.bbc.com/news/technology-55675826
            Possible article: https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html
            Possible article: https://www.cnet.com/news/samsungs-galaxy-s21-upgrades-likely-wont-spell-an-end-to-galaxy-fe-or-note-lines-yet
            Possible article: https://abcnews.go.com/Politics/wireStory/trump-leave-washington-morning-bidens-inauguration-75278801?cid=clicksource_4380645_6_heads_hero_live_headlines_hed

BUT as Erwan pointing out in his comment it's not clear "how you define the difference between article and not article", so my answer is only part of a potential solution to your problem of URL classification.

It's hard to make any solid recommendation for any "out-of-the-box" solution that can meet an undefined use case.

Here are some links that might help you:

I'm still of the mindset that your use case will likely require a multi-prong approach and will need to be refined as more URLs are classified as either "an article or non-article."

$\endgroup$
2
$\begingroup$

There is not going to be a pretrained classifier for article/not article because different people will have different definitions of article/not article.

The first step is define article/not article for your specific use case. That can be done two ways:

  1. Series of rules - List criteria for article/not article, Then encode those rules in a program. For example an article does not contain the phrase "Terms of service". This strategy works well for deterministic, narrow definitions that do not change.

  2. Train a binary classifier - Label thousands of webpages as article/not article. Then preprocess them so they are amenable for machine learning. Then fit a binary classifier, Naive Bayes classifier is a common first choice. This strategy works well for probabilistic, complex definitions that can change.

$\endgroup$
3
  • $\begingroup$ Are you suggesting that the OP build a binary classifier based on the URLs or the content associated with the URLs? $\endgroup$ Jan 14 at 17:34
  • $\begingroup$ I doubt the urls contain enough information. Most likely the binary classifier will have to be trained on the webpages. $\endgroup$ Jan 14 at 19:19
  • $\begingroup$ Thanks for the clarification. Without more details from the OP it's hard to know what types of URLs are within his data set. Those details would help in providing possible solutions to solve his use case. $\endgroup$ Jan 14 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.