Cluster URLs based on their pattern

Question

I am new to clustering techniques and I highly value any input you can provide for my problem bellow. Basically, I want to cluster URLs based on their structural patterns. for example

• cluster1 - simple URLs https://domain/path/file
• cluster2 - shortened URLs
• cluster3 - redirect URLs
• ....
• cluster k - new URL pattern

Given a URL dataset, I want to understand how many different URL pattern clusters exists and then visually see the difference.

What I see in the existing methods are clustering domain wise (cluster URLs of the same website together). And this is not what I am expecting. When I try the nlp based (word based) similarity clustering this is happening as the URLs of the same website tend to have same words with little differences.

Instead, I want to focus on the URL structure and identify URL patterns. Removing all the special characters and just creating a bag of words for each URL nullify the URL structure. Can anyone help me to identify a suitable clustering technique as well as a vectorizing technique to identify different URL pattern clusters.

Thanks in advance Matheesha

Answer 1

There are a few approaches you can take to cluster URLs based on their structural patterns. One option is to use a method called "string distance," which measures the similarity between two strings based on the number of edits required to transform one string into the other. Some common string distance measures include Levenshtein distance and Cosine distance.

To use string distance for clustering, you can first vectorize the URLs by breaking them down into individual tokens (e.g., domain, path, file) and then calculating the string distance between each pair of URLs. You can then use a clustering algorithm such as k-means or hierarchical clustering to group the URLs into clusters based on their string distance.

Another option is to use a method called "regular expression matching," which allows you to define a pattern for the structure of a URL and then match that pattern to a set of URLs. For example, you could use a regular expression to identify URLs with a simple structure (e.g., https://domain/path/file), shortened URLs, and redirect URLs. You could then use a clustering algorithm to group the URLs into clusters based on their match to the different regular expressions.

Hope this helps!