2
$\begingroup$

I have a very basic question about how the k-means clustering algorithm works when the data is stored in HDFS. Does the training take place on the subset of data on each node for a fixed value of K( say 3)? But then we will have 3*number of nodes centroids?

How would it be different if we applied linear regression? Will it only train on data stored on each node locally?

$\endgroup$
  • 1
    $\begingroup$ here is a detailed description: blog.data-miners.com/2008/02/… $\endgroup$ – n1tk Feb 10 '17 at 9:36
  • $\begingroup$ You are welcome. I added the answer bellow so in case of link in the future to be invalid. $\endgroup$ – n1tk Feb 16 '17 at 14:58
0
$\begingroup$

Posting details from the link in comments above.

The data can be partitioned among multiple processors (or streams or threads). Each processor can read the previous iteration's cluster centers and assign the rows on the processor to clusters. Each processor then calculates new centers for its of data. Each actual cluster center (for the data across all processors) is then the weighted average of the centers on each processor.

In other words, the rows of data do not need to be combined globally. They can be combined locally, with the reduced set of results combined across all processors. In fact, MapReduce even contains a "combine" method for just this type of algorithm.

You postulate a shared file that has the centroids as calculated for each processor. This file contains:

  • The iteration number.
  • The cluster id.

  • The cluster coordinates.

  • The number of rows assigned to the cluster.

    This is the centroid file.

An iteration through the algorithm is going to add another set of
rows to this file. This information is the only information that
needs to be communicated globally.

There are two ways to do this in the MapReduce framework. The first uses map, combine, and reduce. The second only uses map and reduce.

link with all details

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.