0
$\begingroup$

I have ~1000 different news websites and I scraped and saved all the internal url links for each website. For instance, the website dcgazette.com has a 2MB text file with associated urls:

1) https://dcgazette.com/writers-wanted/

2) https://dcgazette.com/2019/fifteen-things-that-caught-my-eye-today-june-20-2019/

3) https://dcgazette.com/page/494/

4) https://dcgazette.com/page/3/

5) https://dcgazette.com/2019/trump-has-reason-to-be-concerned-about-2020/

etc

Notice that urls 2 and 5 are associated with actual articles. Across websites, each has a different way of representing their article urls. But within each website, all article urls have the same format, i.e. "domain/date/title" (for dcgazette.com).

Is there any unsupervised learning algorithm to process these url strings such that I can cluster all article urls together for each domain? The result I am aiming for is extracting all article urls for each domain with as little noise as possibe from non-article urls for all these domains.

An idea I had was separating each url into pieces by splitting on "/" and generating features such as length or type of letters used for each piece and using a k-means clustering algorithm. I'm not sure how this will actually perform however and am open to advice on any methodology that can make this as accurate as possible

Improve this question
$\endgroup$
1
  • $\begingroup$ Unsupervised methods will depend on you choosing appropriate features here. But that means you already solved the problem then: recognize typical parts of URLs, such as dates. You are then better off just doing this directly and exactly rather than hoping that clustering may - or may not!!! - find such patterns... $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Jun 30, 2019 at 21:25

2 Answers 2

Reset to default
1
$\begingroup$

I don't think you need any ML, in the best case it's going to be very slow compared to direct processing and it's very likely to cause errors.

For each address you can parse the url string and extract the domain substring (in your example they are all prefixes, if it's always the case it's very easy). Then you just group them by domain in a map... done.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I'm trying to find a general way to differentiate between article urls and non article urls for all news domains. I cannot guarantee that every news domain will have article url in "domain/date/title" format. For dcgazette, I agree that I can use a regex that is specific to this domain to distinguish between article urls and non article urls. But there are websites that have url in formats that are "domain/random_integer_string", or whatever you can think of, that do not have this format and thus they need their own specific regex. If I do it this way, I will need to write 1000 regexes. $\endgroup$
    – user76528
    Commented Jun 26, 2019 at 6:00
  • $\begingroup$ I can say more generally that if I look at all urls of a news website, all article urls from that website will be in the same format. I was hoping that there exists some clustering algorithm that can cluster urls of similar formats together, and then using the number of urls in that cluster or some other heuristic I can choose the cluster that contains article urls $\endgroup$
    – user76528
    Commented Jun 26, 2019 at 6:06
  • $\begingroup$ urls follow specific rules so imho there's little point trying to re-learn these rules with ML, that will only introduce errors. this is a common task, google "parse domain from url <language>" and you're likely to find reliable methods to extract domains. Example for python: quora.com/How-do-I-extract-only-the-domain-name-from-an-URL $\endgroup$
    – Erwan
    Commented Jun 26, 2019 at 9:48
  • $\begingroup$ Maybe I'm being misunderstood but I'm not looking at differentiating between domains. I'm looking within a domain, trying to differentiate article urls from non article urls, and need to do this for all domains in my corpus. I already use tldextract to group my urls by domains. Now the question is within each domain, how do I get all article urls and filter out those that are not associated with articles $\endgroup$
    – user76528
    Commented Jun 26, 2019 at 15:48
  • $\begingroup$ ooooh ok, I got your question completely wrong then. in my defense it was a bit misleading, I thought "cluster all article urls together for each domain" meant "cluster by domain". $\endgroup$
    – Erwan
    Commented Jun 26, 2019 at 17:18
1
$\begingroup$

agree with Erwan! Let say you can use regex in loop to extract domain, date and article title. Let say you are doing it in python. Then in order to extract title you need to run the following code: 

import re
string = 'https://dcgazette.com/2019/trump-has-reason-to-be-concerned-about-2020/'

title = re.search('/[a-zA-Z0-9-]*/$', string)
print(title.group(0).replace('/', ''))
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.