How can I reduce the number of dimensions using a Clustering algorithm in a mixed dataset?

Question

I am working with a mixed data set, corresponding to TV consumption data, with the aim of reducing the number of features to only those relevant to detect TV consumption patterns (or consumption groups) using clustering.

The dataset is composed of about 20 dimensions and 2.000.000 samples for 1 day of consumption. I have access to the data of up to 3 years of consumption, so I can exploit up to ~1 billion data. My idea is to start working with only a few days of consumption (and therefore a few million data).

3 dimensions are of continuous/numerical type (the date-time of consumption, the duration...) and the remaining dimensions are of discrete/categorical type, with features with binary options (e.g. whether the programme is live or not) or with multiple and even hundreds of options (e.g. name of the programme, theme, type of device, etc). For this reason, I am trying to implement a clustering algorithm with Python that can deal with mixed data.

Since I suspect that there are many dimensions that might be irrelevant to my study, I would like to reduce the number of features. To do so, I have thought of applying some clustering algorithm and check that, by removing a given feature, the clustering results are not affected.

However, I don't know what metric I should calculate or use to evaluate the clustering results and the effect of removing one of the attributes.

At the moment I have applied the K-prototypes algorithm which is based on K-means but for mixed data. It is easy to implement in Python (https://antonsruberts.github.io/kproto-audience/). However, it only returns the cluster labels, the centroid coordinates and the cost (defined as the sum of the distance of all points to their respective centroids). Therefore, I do not know how to interpret the results or how to study the effect of the features.

I would like to know if my approach to the problem is correct and what metric I should use to evaluate the clustering results and reduce the number of dimensions, as well as if there are other easily implemented algorithms in Python (for clustering or unsupervised feature selection) capable of dealing with the type of data I have.

Answer 1

Some common techniques to reduce number of features:

• Missing Values Ratio. Data columns with too many missing values are unlikely to carry much useful information. Thus data columns with number of missing values greater than a given threshold can be removed. The higher the threshold, the more aggressive the reduction.

• Low Variance Filter. Similarly to the previous technique, data columns with little changes in the data carry little information. Thus all data columns with variance lower than a given threshold are removed. A word of caution: variance is range dependent; therefore normalization is required before applying this technique.

• Random Forests / Ensemble Trees. Decision Tree Ensembles, also referred to as random forests, are useful for feature selection in addition to being effective classifiers. Personally I will prefer this method as it is easy to implement. Example use is given in the kernel --> https://www.kaggle.com/prashant111/xgboost-k-fold-cv-feature-importance?scriptVersionId=48823316&cellId=74

• High Correlation Filter. Data columns with very similar trends are also likely to carry very similar information. In this case, only one of them will suffice to feed the machine learning model. Here we calculate the correlation coefficient between numerical columns and between nominal columns as the Pearson’s Product Moment Coefficient and the Pearson’s chi square value respectively. Pairs of columns with correlation coefficient higher than a threshold are reduced to only one. A word of caution: correlation is scale sensitive; therefore column normalization is required for a meaningful correlation comparison.

Refer : https://www.kdnuggets.com/2015/05/7-methods-data-dimensionality-reduction.html