I'm new to apache spark.
Is it possible to configure multi cluster spark without hadoop?
If so, can you please provide the steps. I would like to create clusters on Google Compute Engine (1-master, 1-worker)
4
-
$\begingroup$ Spark Cluster on Google Compute Engine $\endgroup$– EmreDec 7, 2014 at 23:04
-
$\begingroup$ @Emre, I've also looked into that blog, But there are no steps provided for cluster formation. It is single node. $\endgroup$– user4290511Dec 9, 2014 at 6:20
-
$\begingroup$ You indicated that you wanted 1 slave, so setting the second parameter to 1 should do the trick. $\endgroup$– EmreDec 9, 2014 at 6:31
-
$\begingroup$ What do you want to use for a file system and cluster manager instead of Hadoop? Options include Cassandra for files, Mesos or Yarn for cluster management, etc. $\endgroup$– nealmcbApr 16, 2015 at 19:28
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
As explained in the comments by @Emre, there is a python script on github which you can use to set up a cluster of whatever size you like: Spark Cluster on Google Compute Engine | Ido Green. It allows you to specify the instance type to use for the workers (slaves) and separately what to use for the master.
Note that it uses gcutil, which is deprecated in favor of gcloud compute
