I am new to clustering techniques and I highly value any input you can provide for my problem bellow. Basically, I want to cluster URLs based on their structural patterns. for example

  • cluster1 - simple URLs https://domain/path/file
  • cluster2 - shortened URLs
  • cluster3 - redirect URLs
  • ....
  • cluster k - new URL pattern

Given a URL dataset, I want to understand how many different URL pattern clusters exists and then visually see the difference.

What I see in the existing methods are clustering domain wise (cluster URLs of the same website together). And this is not what I am expecting. When I try the nlp based (word based) similarity clustering this is happening as the URLs of the same website tend to have same words with little differences.

Instead, I want to focus on the URL structure and identify URL patterns. Removing all the special characters and just creating a bag of words for each URL nullify the URL structure. Can anyone help me to identify a suitable clustering technique as well as a vectorizing technique to identify different URL pattern clusters.

Thanks in advance Matheesha


1 Answer 1


There are a few approaches you can take to cluster URLs based on their structural patterns. One option is to use a method called "string distance," which measures the similarity between two strings based on the number of edits required to transform one string into the other. Some common string distance measures include Levenshtein distance and Cosine distance.

To use string distance for clustering, you can first vectorize the URLs by breaking them down into individual tokens (e.g., domain, path, file) and then calculating the string distance between each pair of URLs. You can then use a clustering algorithm such as k-means or hierarchical clustering to group the URLs into clusters based on their string distance.

Another option is to use a method called "regular expression matching," which allows you to define a pattern for the structure of a URL and then match that pattern to a set of URLs. For example, you could use a regular expression to identify URLs with a simple structure (e.g., https://domain/path/file), shortened URLs, and redirect URLs. You could then use a clustering algorithm to group the URLs into clusters based on their match to the different regular expressions.

Hope this helps!

  • $\begingroup$ Mohith7548 Thank you very much for the answer. I feel your approaches are right for me. I am just not clear on calculating the string distance. Given there are10k URL dataset, how can I decide on the centroids for each URL token to calculate the distances $\endgroup$
    – Mathee
    Dec 29, 2022 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.