I am new to clustering techniques and I highly value any input you can provide for my problem bellow. Basically, I want to cluster URLs based on their structural patterns. for example
- cluster1 - simple URLs https://domain/path/file
- cluster2 - shortened URLs
- cluster3 - redirect URLs
- cluster k - new URL pattern
Given a URL dataset, I want to understand how many different URL pattern clusters exists and then visually see the difference.
What I see in the existing methods are clustering domain wise (cluster URLs of the same website together). And this is not what I am expecting. When I try the nlp based (word based) similarity clustering this is happening as the URLs of the same website tend to have same words with little differences.
Instead, I want to focus on the URL structure and identify URL patterns. Removing all the special characters and just creating a bag of words for each URL nullify the URL structure. Can anyone help me to identify a suitable clustering technique as well as a vectorizing technique to identify different URL pattern clusters.
Thanks in advance Matheesha