I want to use DBSCAN to recognize any clusters within all text elements from the DOM tree of any webpage. For example all menu items shall be clustered separatey to all main content or footer elements.

My features for now are: - DOM path to the text element - visual rendered CSS properties (used PhantomJS) of the text element - the middle x and y position of the text element - the width of the text element

Before using DBSCAN I use StandardScaler for my feature vector.

But I have no good results even after trying channging eps.

Now the question: Can DBSCAN handle att all such different kind of features like OneHotEncoding for DOM path combined (!) with features like positions or widths in pixel? Should I weight some features before using StandardScaler that all have a similar value?


Automatic weighting will likely not be enough.

For examples standard scaler will assign twice as much weight to the one-hot encoded parts than to the other attributes. Plus, it is based on the assumption that all variables should have the same (assuming they are all normal distributed) weight. But should they all have the same weight? Does it even make sense to use distances of one-hot encoded data?

You are not ready to use DBSCAN (or any other clustering) if you don't have a reliable distance yet that discriminates similar from dissimilar instances.

| improve this answer | |
  • $\begingroup$ Does this mean that all features (=dimensional) should have the same scale or reference System? For example all Features Must have meters. Or is it possible that some Features are e.g. meters, some are colors, and some others are ages. $\endgroup$ – jochen6677 Jan 1 '19 at 10:48
  • $\begingroup$ There is no simple answer. Even if they are all meters they can be different in meaning. A person's location with GPS has 5-10 meters accuracy, so a difference of two meters is not meaningful. The height of persons usually does not very by two meters. So combining these two attributes requires careful considerations that depend on the problem to solve. $\endgroup$ – Has QUIT--Anony-Mousse Jan 1 '19 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.