2
$\begingroup$

I am attempting to train a random forest classifier (pyspark.ml.classification.RandomForestClassifier) on a large dataset (~70gb). However, I am not sure what to send to each of featuresCol, labelCol, predictionCol, and probabilityCol.

From the docs I gather that:

  1. featuresCol is a list of features from your dataframe
  2. labelCol is the target feature
  3. predictionCol is also the target feature, but generated by the model (not sure). Do I need to set this prior to training?
  4. probabilityCol is the probability of each class as a vector. Is this similar to sklearn's class_weight? ie does the model account for a low diversity? If so how?

Additionally, can I set an option for OOB_score?

clf = RandomForestClassifier(featuresCol=feature_cols, labelCol=target_col, numTrees=300, MaxDepth=15, Impurity='gini', maxMemoryInMB=2**10)
clf_t = clf.fit(train)

y_train_pred = clf_t.transform(test)
y_test_pred = clf_t.transform(test)

Here's a link to the docs: https://spark.apache.org/docs/2.2.0/api/python/pyspark.ml.html#module-pyspark.ml.classification

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.