I have this huge mixed data set consisting of both numerical and categorical attributes which upon OneHotEncoding results into a data set with very high dimensionality.

Is it wise to apply machine learning algorithms like K-means clustering, dimensionality reduction and regression on subsets of data set? For example applying K-means clustering to numerical columns first and join the result with categorical data set later.

  • $\begingroup$ Not at all...If you can accomodate everything in your Memory, then its fine else Sorry you don't have any option.. Rather try to search for different algorithms/libraries/packages which can handle such data-sets automatically for you $\endgroup$
    – Aditya
    Feb 28, 2018 at 14:13

1 Answer 1


Applying a machine learning algorithm on only a subset of the data and including other subsets later does not allow the algorithm to assess the importance of each attribute equally.

For example, say you have a data set called A, which has subsets B and C. Without loss of genearality, if you fit a model ('apply an algorithm') on subset B, and then include subset C later, then you're saying 'given subset B is already in the model, assess the impact of including subset C'. Instead, if you apply the entire algorithm on the entire data set (A), then you're allowing the algorithm to discover which features are most important for the desired outcome.

That being said, it may be wise to process the different elements of your data set differently. That is, categorical covariates may be modelled differently from continuous covariates. If you're using something like a feed-forward neural network, then it's not a big deal, but if you're using a more traditional statistical model you may need to take that into account. For example, in R, you need to specify that a categorical covariate is in fact a 'factor' variable.

  • $\begingroup$ Thanks @StatsSorceress. I am using PySpark right now. Is there a way to encode categorical values to numerical and not increasing the dimensionality so much? Because OneHotEncoding can result in 2000 more columns. $\endgroup$
    – moirK
    Mar 1, 2018 at 9:20
  • $\begingroup$ Hi @zimmer, I'm not a PySpark user, so perhaps that would be a better question for StackOverflow. However, if you have more than 2000 categories, that makes me wonder if that variable is truly categorical. $\endgroup$ Mar 1, 2018 at 13:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.