There is a project that you may be interested in called Koalas which allows running pandas on top of Spark.
Koalas: pandas API on Apache Spark The Koalas project makes data scientists more productive when interacting with big data, by
implementing the pandas DataFrame API on top of Apache Spark. pandas
is the de facto standard (single-node) DataFrame ...
For some reason in cross validation we also need to set the label column of the evaluator (even tho it's already set for the estimator. So all you need to do is change BinaryClassificationEvaluator() into BinaryClassificationEvaluator().setLabelCol("churn") where "churn" is the name of your target variable.