4
$\begingroup$

I have a dataset of xyz coordinates with a date component in a pandas dataframe

ex:

  • date1: $[x_1,y_1,z_1]$,
  • date2: $[x_2,y_2,z_2]$,
  • date3: $[x_3,y_3,z_3]$, ..

I would like to classify a sample of object positions over the period of a week (using indexes to re-map the classification label back to the date), like this:

  • Week 1: $[x_1,y_1,z_1], [x_2, y_2, z_2], [x_3,y_3,z_3], [x_4,y_4,z_4], [x_5,y_5,z_5], [x_6,y_6,z_6], [x_7,y_7,z_7]$,
  • Week 2: $[x_8,y_8,z_8],[x_9,y_9,z_9],[x_{10},y_{10},z_{10}],[x_{11},y_{11},z_{11}],[x_{12},y_{12},z_{12}],[x_{13},y_{13},z_{13}],[x_{14},y_{14},z_{14}]$,

When I try to run KMeans it returns

k_means = KMeans(n_clusters=cclasses)
k_means.fit(process_set.hpc)
date_classes = k_means.labels_

ValueError: Found array with dim 3. Expected <= 2

Questions:

  • Do I have to run it through Principal Component Analysis (PCA) first? if so, how do I maintain date mapping to the classification created?
  • Are there any other methods I could use?
  • Am I doing everything completely backwards and should consider a different approach, any thoughts?

Thanks!

Improve this question
$\endgroup$
3
  • $\begingroup$ Do you consider your data set time series? $\endgroup$
    – Aleksandr Blekh
    Jan 11, 2015 at 0:08
  • 2
    $\begingroup$ If you want to classify data, you should not use a clustering method. $\endgroup$
    – bayer
    Jan 13, 2015 at 10:30
  • $\begingroup$ Please include a printout of process_set and process_set.hpc (the first few lines as printed by pandas are enough). Also, please clarify if you want to cluster or classify this dataset. You cannot classify if you do not have pre-existing labels. $\endgroup$
    – logc
    Jan 14, 2015 at 16:55

2 Answers 2

Reset to default
3
$\begingroup$

I don't have enough reputation to ask questions in comment for clarification before answering it, so I'm going to do both here.

Here are the things that would help answering this question for now:

Can you post part of the process_set.hpc?

What's its format? Is it a numpy array? Is it a Pandas dataframe?

What's the value of cclasses?

And now the answer:

First of all, k-means algorithm is able to find clusters in any n-dimensional data. If n is too big, it is better to use PCA but for n=3 that wouldn't necessarily add any value.

The second thing that looks suspicious to me is that in the documentation for kmeans in scikit-learn, there is no compute_labels option, as seen here. However, that option exists for MiniBatchKMeans as seen here.

Also, if you make your data in the form of a pandas dataframe (if it is not already so), things would be much easier to track and you won't have to reattach the timing information to your data afterwards.

I may be able to give you a more thorough answer if I know a bit more about the format of the data.

Good luck!

Improve this answer
$\endgroup$
0
$\begingroup$

If you can provide more details about the processing you're doing to the data, I think the responses would be a little more helpful.

Here are a few things to consider:

  • What's the shape of your data going into k-means? Are you aggregating up to the week level? If so, then your data won't look like the example you posted, since that seems like daily data.

  • Try using .shape on your output data set. Does the # of rows match the data set you put into it? (This ties in to question #1. You'll be unable to join back directly if your original data is daily and the k-means data is weekly.

  • I'm not sure if this is what you're attempting, but just declaring data_classes as the labels isn't going to add them back in to your original data. Like oxtay suggested, using pandas is a good choice because it will allow you to join.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.