4
$\begingroup$

Consider an unsupervised data. The data is in the form of a csv file( I am using pandas dataframe for this). Its a web page data at different time steps and the way I am converting data to be fed to my model(K-means) is by taking difference of the time steps of the current web-Page ID load to next web_page ID load.

Now, there are some features in the data like "scroll" (which represents a human scrolling on that webpage) which is occurring multiple times for the same web page ID. Since I am only using delta the way I want to encode this "scroll" as a feature is how many times it happened between the delta(time difference). This gives the frequency.

Now the question is should I do some processing on this raw frequency I calculated, or can I directly feed it to my model. In case more processing is needed, what do you suggest?

$\endgroup$
1
$\begingroup$

As generic advice, any algorithm that uses distances might be affected by scaling or normalization. Thus, if you use one of them, you need to think about it twice and consider how this will alter your problem.

I will take the answer from the CrossValidated StackExchange since another user has already explained it there.

If your variables are of incomparable units (e.g. height in cm and weight in kg) then you should standardize variables, of course. Even if variables are of the same units but show quite different variances it is still a good idea to standardize before K-means. You see, K-means clustering is "isotropic" in all directions of space and therefore tends to produce more or less round (rather than elongated) clusters. In this situation leaving variances unequal is equivalent to putting more weight on variables with smaller variance, so clusters will tend to be separated along variables with greater variance.

ttnphns's answer

$\endgroup$
  • $\begingroup$ Agreed. Could you also provide insights on how to normalize frequency data. I have four column of frequency data and its very sparse. $\endgroup$ – Heisenbug Feb 20 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.