I'm trying to set up a cluster (1 namenode, 1 datanode) on AWS. I'm using free one year trial period of AWS, but the challenge is, instance is created with 1GB of RAM.

As I'm a student, I cannot afford much. Can anyone please suggest me some solution?

Also, it would be great if you could provide any links for setting up multi cluster hadoop with spark on AWS.

Note: I cannot try in GCE as my trial period is exhausted.

  • 1
    $\begingroup$ Install it on your own computer. It has more than a gig right? $\endgroup$ – Emre Dec 7 '14 at 12:00
  • $\begingroup$ yep, but my system has only 4GB of RAM. I tried using ubuntu in VM. It's very low. $\endgroup$ – user4290511 Dec 7 '14 at 12:59
  • 2
    $\begingroup$ Talk to your professors and tell them you need a bigger machine to use. Most CS departments will help you out. $\endgroup$ – sheldonkreger Dec 10 '14 at 1:39
  • $\begingroup$ You can always create another google account and use GCE from that. $\endgroup$ – DePP Feb 28 '15 at 6:16

So if 4GB of RAM isn't sufficient, 1GB isn't going to be. That is really too little to run an HDFS namenode, a datanode, YARN, Spark driver alone, let alone leaving room for your workers.

Much more reasonable is to simply run Spark locally on that instance without Hadoop at all.

But I would question whether Spark is the right choice if you are definitely limited to such a small machine.

| improve this answer | |
  • $\begingroup$ yes in fact i have the same questions in my mind. No other choice left except setting up a physical machine with good amount of memory or creating a large instance in AWS. $\endgroup$ – user4290511 Dec 7 '14 at 13:49
  • $\begingroup$ Or you could upgrade your computer; memory is cheap. $\endgroup$ – Emre Dec 7 '14 at 23:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.