4
$\begingroup$

I'm new to apache spark.

Is it possible to configure multi cluster spark without hadoop?

If so, can you please provide the steps. I would like to create clusters on Google Compute Engine (1-master, 1-worker)

$\endgroup$
4
  • $\begingroup$ Spark Cluster on Google Compute Engine $\endgroup$
    – Emre
    Dec 7, 2014 at 23:04
  • $\begingroup$ @Emre, I've also looked into that blog, But there are no steps provided for cluster formation. It is single node. $\endgroup$ Dec 9, 2014 at 6:20
  • $\begingroup$ You indicated that you wanted 1 slave, so setting the second parameter to 1 should do the trick. $\endgroup$
    – Emre
    Dec 9, 2014 at 6:31
  • $\begingroup$ What do you want to use for a file system and cluster manager instead of Hadoop? Options include Cassandra for files, Mesos or Yarn for cluster management, etc. $\endgroup$
    – nealmcb
    Apr 16, 2015 at 19:28

1 Answer 1

1
$\begingroup$

As explained in the comments by @Emre, there is a python script on github which you can use to set up a cluster of whatever size you like: Spark Cluster on Google Compute Engine | Ido Green. It allows you to specify the instance type to use for the workers (slaves) and separately what to use for the master.

Note that it uses gcutil, which is deprecated in favor of gcloud compute

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.