Suppose you want to cluster, or classify, web pages inside a domain. In the same domain similar web pages always have the same structure more or less. (think of every product page an e-commerce would have)

My idea is to convert the HTML tags of every page to a numeric vector, then use those with the clustering method of your choice. In this contest we would not need the actual content of the page, just the structure.

But how to do that?

The first idea I got was to assign every tag a position (sequential), a depth (how inside of other tags it is), and an ID that would identify which kind of tag it is (< img >, < h1 >, < div >...)

  • $\begingroup$ Well, this step is the one where you need a good idea then... Turning data into a vector and then magically solving everything is not really an "idea", you know. I have major doubt it will be sufficient to only look at the structure; because the HTML structure is known to change, even when the page contents remain the same... $\endgroup$ – Has QUIT--Anony-Mousse Sep 4 '19 at 13:25
  • $\begingroup$ If a web page structure is going to change, every page of the same type most of the times will have the same changes. Especially for websites created with a CMS or something similar $\endgroup$ – Nicolò Gasparini Sep 5 '19 at 14:48
  • $\begingroup$ So everything will have the same vector, sounds useless to me? $\endgroup$ – Has QUIT--Anony-Mousse Sep 5 '19 at 15:06
  • $\begingroup$ But the goal is to cluster together pages with different urls but the same html structure $\endgroup$ – Nicolò Gasparini Sep 5 '19 at 16:09
  • $\begingroup$ Well, if all you care is the structure, then just go ahead and define structural features, then cluster them. But if they are that similar and come from a CMS, I am fairly certain there is an easier way to do this. Such as product identifiers. $\endgroup$ – Has QUIT--Anony-Mousse Sep 6 '19 at 0:34

I'm not sure a vector is the best way to represent a web page. Given the tree structure, you love valuable information if you "transform" if into a vector. Given how different one web page can be to another, you might end up with a very large dimension feature vector.

Maybe graphs are better suited to your problem?

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.