I'm trying to set up a cluster (1 namenode, 1 datanode) on AWS. I'm using free one year trial period of AWS, but the challenge is, instance is created with 1GB of RAM.

As I'm a student, I cannot afford much. Can anyone please suggest me some solution?

Also, it would be great if you could provide any links for setting up multi cluster hadoop with spark on AWS.

Note: I cannot try in GCE as my trial period is exhausted.

  • 1
    $\begingroup$ Install it on your own computer. It has more than a gig right? $\endgroup$
    – Emre
    Dec 7, 2014 at 12:00
  • $\begingroup$ yep, but my system has only 4GB of RAM. I tried using ubuntu in VM. It's very low. $\endgroup$ Dec 7, 2014 at 12:59
  • 2
    $\begingroup$ Talk to your professors and tell them you need a bigger machine to use. Most CS departments will help you out. $\endgroup$ Dec 10, 2014 at 1:39
  • $\begingroup$ You can always create another google account and use GCE from that. $\endgroup$
    – DePP
    Feb 28, 2015 at 6:16

1 Answer 1


So if 4GB of RAM isn't sufficient, 1GB isn't going to be. That is really too little to run an HDFS namenode, a datanode, YARN, Spark driver alone, let alone leaving room for your workers.

Much more reasonable is to simply run Spark locally on that instance without Hadoop at all.

But I would question whether Spark is the right choice if you are definitely limited to such a small machine.

  • $\begingroup$ yes in fact i have the same questions in my mind. No other choice left except setting up a physical machine with good amount of memory or creating a large instance in AWS. $\endgroup$ Dec 7, 2014 at 13:49
  • $\begingroup$ Or you could upgrade your computer; memory is cheap. $\endgroup$
    – Emre
    Dec 7, 2014 at 23:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.