I'm new to apache spark.

Is it possible to configure multi cluster spark without hadoop?

If so, can you please provide the steps. I would like to create clusters on Google Compute Engine (1-master, 1-worker)

  • $\begingroup$ Spark Cluster on Google Compute Engine $\endgroup$ – Emre Dec 7 '14 at 23:04
  • $\begingroup$ @Emre, I've also looked into that blog, But there are no steps provided for cluster formation. It is single node. $\endgroup$ – user4290511 Dec 9 '14 at 6:20
  • $\begingroup$ You indicated that you wanted 1 slave, so setting the second parameter to 1 should do the trick. $\endgroup$ – Emre Dec 9 '14 at 6:31
  • $\begingroup$ What do you want to use for a file system and cluster manager instead of Hadoop? Options include Cassandra for files, Mesos or Yarn for cluster management, etc. $\endgroup$ – nealmcb Apr 16 '15 at 19:28

As explained in the comments by @Emre, there is a python script on github which you can use to set up a cluster of whatever size you like: Spark Cluster on Google Compute Engine | Ido Green. It allows you to specify the instance type to use for the workers (slaves) and separately what to use for the master.

Note that it uses gcutil, which is deprecated in favor of gcloud compute

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.