# multi-label prediction with pySpark

I am new to Spark

I am using pyspark to predict a multi label results.

I have converted multi labels to binary

So my labels will look like this

Red  Green  Blue  Yellow  Black  White  Brown  Pink  Orange
1    1      1     0       1      0      1      0     0
0    0      0     1       0      0      0      0     1
1    1      0     0       1      1      1      1     0
1    1      0     1       1      0      0      1     0
0    0      1     0       0      0      0      0     0
0    1      1     0       1      0      1      1     1


my spark dataframe looks like this

vhouse_df.show(3)
+--------------------+---------------+
|            features|         labels|
+--------------------+---------------+
|[21.0,5.0,5.0,21....|(31,[30],[1.0])|
|[0.0,3.0,3.0,0.0,...|(31,[30],[1.0])|
|[32.0,8.0,8.0,32....|(31,[30],[1.0])|
+--------------------+---------------+


I found this question in StackOverFlow https://stackoverflow.com/questions/33551747/logistic-regression-pyspark-mllib-issue-with-multiple-labels

Tried to do the same but got error

Here is my code

from pyspark.mllib.classification import LogisticRegressionWithLBFGS  #<----- (1)
lr = LogisticRegressionWithLBFGS.train(sc.parallelize(train_df))     #<----- (2)
lr_model = lr.fit(train_df)                                           #<----- (3)
print("Coefficients: " + str(lr_model.coefficients))                  #<----- (4)
print("Intercept: " + str(lr_model.intercept))                        #<----- (5)


Line (2) causing this error

lr = LogisticRegressionWithLBFGS.train(sc.parallelize(train_df))
Traceback (most recent call last):

File "<ipython-input-11-05522f46c27a>", line 1, in <module>
lr = LogisticRegressionWithLBFGS.train(sc.parallelize(train_df))

File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\context.py", line 527, in parallelize
jrdd = self._serialize_to_jvm(c, serializer, reader_func, createRDDServer)

File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\context.py", line 559, in _serialize_to_jvm
serializer.dump_stream(data, tempFile)

File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\serializers.py", line 352, in dump_stream
self.serializer.dump_stream(self._batched(iterator), stream)

File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\serializers.py", line 143, in dump_stream
self._write_with_length(obj, stream)

File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\serializers.py", line 153, in _write_with_length
serialized = self.dumps(obj)

File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\serializers.py", line 583, in dumps
return pickle.dumps(obj, protocol)

File "C:\Users\GX\anaconda3\lib\site-packages\py4j\java_gateway.py", line 1257, in __call__

File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\sql\utils.py", line 63, in deco
return f(*a, **kw)

File "C:\Users\GX\anaconda3\lib\site-packages\py4j\protocol.py", line 332, in get_return_value
format(target_id, ".", name, value))

Py4JError: An error occurred while calling o102.__getstate__. Trace:
py4j.Py4JException: Method __getstate__([]) does not exist
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
at py4j.Gateway.invoke(Gateway.java:274)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)

• Try changing sc.parallelize(train_df) to train_df.rdd in the second line and convert each row to LabeledPoint. Also, the third line should also use the same converted RDD, not a dataframe. Jun 9 '20 at 5:56