I have ~1000 different news websites and I scraped and saved all the internal url links for each website. For instance, the website dcgazette.com has a 2MB text file with associated urls:
Notice that urls 2 and 5 are associated with actual articles. Across websites, each has a different way of representing their article urls. But within each website, all article urls have the same format, i.e. "domain/date/title" (for dcgazette.com).
Is there any unsupervised learning algorithm to process these url strings such that I can cluster all article urls together for each domain? The result I am aiming for is extracting all article urls for each domain with as little noise as possibe from non-article urls for all these domains.
An idea I had was separating each url into pieces by splitting on "/" and generating features such as length or type of letters used for each piece and using a k-means clustering algorithm. I'm not sure how this will actually perform however and am open to advice on any methodology that can make this as accurate as possible