0
$\begingroup$

I am working with a dataset where each elements is a square table of size m-by-n, where m (the number of rows) is the same for all the data points, while n (the number of columns) varies from tens to thousands. I need to classify the elements of this dataset in two or more clusters (or alternatively, determine outlier/atypical elements.)

What I am looking for is:

  • either ML/statistical algorithms adapted for working with such disparate size datasets
  • or typical transformations reducing the elements of the dataset to a common feature space (one possibility is calculating correlations of rows, thus representing each element by an m-by-m matrix.)

I apologize, if the question is too vague or not suitable for this community. I would however appreciate any tips/ideas.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Do you expect the size of the dataset to be highly correlated with the classification into clusters? If so, then it could be a straightforward approach to simply count the number of columns and use that as a feature to a K-means clustering algorithm. If you think that the size is not related to the cluster that a particular data point should be assigned to, then I would recommend something like PCA to reduce the number of dimensions of each data point: https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html

 pca = PCA(n_components=2)
 pca.fit(X)

You might end up having to fit each data point separately, but it would be a fast/simple approach to start. You could also combine PCA with counting the number of original dimensions potentially.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Do you mean using PCoA on each data point and then using only the first two components as a representation? $\endgroup$
    – Roger V.
    Sep 22 at 7:42
  • $\begingroup$ Yes, sorry it wasn't clear. Could be more than 2 if that's required to capture the variance in the dataset, but I usually start with a small number and work upwards. Also, using 2 components makes it particularly easy to visualize. $\endgroup$
    – brewmaster321
    Sep 22 at 7:51
  • $\begingroup$ I understand that it could be more than 2 - just checking that I correctly understood the core of the approach. Thanks. $\endgroup$
    – Roger V.
    Sep 22 at 7:54
  • $\begingroup$ Actually thinking about this a bit more now that coffee has kicked in, the PCA approach might not work - if every data point has a different set of singular values and there's no commonality in the ordering of the dimensions, the singular vectors that you get back from PCA could point in different directions for each data point. Can you elaborate a bit more on the data itself, and what a data point represents? $\endgroup$
    – brewmaster321
    Sep 22 at 9:00
  • $\begingroup$ The rows all correspond to the same set of features, and ordered in the same way - so I think your approach should work. $\endgroup$
    – Roger V.
    Sep 22 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.