I am attempting to train a random forest classifier (pyspark.ml.classification.RandomForestClassifier
) on a large dataset (~70gb). However, I am not sure what to send to each of featuresCol, labelCol, predictionCol, and probabilityCol.
From the docs I gather that:
featuresCol
is a list of features from your dataframelabelCol
is the target featurepredictionCol
is also the target feature, but generated by the model (not sure). Do I need to set this prior to training?probabilityCol
is the probability of each class as a vector. Is this similar to sklearn'sclass_weight
? ie does the model account for a low diversity? If so how?
Additionally, can I set an option for OOB_score?
clf = RandomForestClassifier(featuresCol=feature_cols, labelCol=target_col, numTrees=300, MaxDepth=15, Impurity='gini', maxMemoryInMB=2**10)
clf_t = clf.fit(train)
y_train_pred = clf_t.transform(test)
y_test_pred = clf_t.transform(test)
Here's a link to the docs: https://spark.apache.org/docs/2.2.0/api/python/pyspark.ml.html#module-pyspark.ml.classification