3
$\begingroup$

On the node from which I run Dumbo commands, all the files produced as output are produced on the same node. For example, suppose there is a node having name hvs on which I ran the script:

dumbo start matrix2seqfile.py -input hdfs://hm1/user/trainf1.csv -output hdfs://hm1/user/train_hdfs5.mseq -numreducetasks 25 -hadoop $HADOOP_INSTALL

When I inspect my file system, I find that all the files produced are accumulated only in the hvs node.

Ideally, I'd like the files to get distributed throughout the cluster--my data is not getting balanced throughout the cluster. Can anyone advise me on how to fix this?

$\endgroup$
1
$\begingroup$

In hadoop etc folder,hdfs-site.xml was having number of replication units as 1.That is why,all the files were getting saved on one single node.I changed it and the problem resolved.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.