0
$\begingroup$

I am new to Spark

I am using pyspark to predict a multi label results.

I have converted multi labels to binary

So my labels will look like this

Red  Green  Blue  Yellow  Black  White  Brown  Pink  Orange
1    1      1     0       1      0      1      0     0
0    0      0     1       0      0      0      0     1
1    1      0     0       1      1      1      1     0
1    1      0     1       1      0      0      1     0
0    0      1     0       0      0      0      0     0
0    1      1     0       1      0      1      1     1

my spark dataframe looks like this

vhouse_df.show(3)
+--------------------+---------------+
|            features|         labels|
+--------------------+---------------+
|[21.0,5.0,5.0,21....|(31,[30],[1.0])|
|[0.0,3.0,3.0,0.0,...|(31,[30],[1.0])|
|[32.0,8.0,8.0,32....|(31,[30],[1.0])|
+--------------------+---------------+

I found this question in StackOverFlow https://stackoverflow.com/questions/33551747/logistic-regression-pyspark-mllib-issue-with-multiple-labels

Tried to do the same but got error

Here is my code

from pyspark.mllib.classification import LogisticRegressionWithLBFGS  #<----- (1)
lr = LogisticRegressionWithLBFGS.train(sc.parallelize(train_df))     #<----- (2)
lr_model = lr.fit(train_df)                                           #<----- (3)
print("Coefficients: " + str(lr_model.coefficients))                  #<----- (4)
print("Intercept: " + str(lr_model.intercept))                        #<----- (5)

Line (2) causing this error

lr = LogisticRegressionWithLBFGS.train(sc.parallelize(train_df))
Traceback (most recent call last):

  File "<ipython-input-11-05522f46c27a>", line 1, in <module>
    lr = LogisticRegressionWithLBFGS.train(sc.parallelize(train_df))

  File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\context.py", line 527, in parallelize
    jrdd = self._serialize_to_jvm(c, serializer, reader_func, createRDDServer)

  File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\context.py", line 559, in _serialize_to_jvm
    serializer.dump_stream(data, tempFile)

  File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\serializers.py", line 352, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)

  File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\serializers.py", line 143, in dump_stream
    self._write_with_length(obj, stream)

  File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\serializers.py", line 153, in _write_with_length
    serialized = self.dumps(obj)

  File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\serializers.py", line 583, in dumps
    return pickle.dumps(obj, protocol)

  File "C:\Users\GX\anaconda3\lib\site-packages\py4j\java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)

  File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\sql\utils.py", line 63, in deco
    return f(*a, **kw)

  File "C:\Users\GX\anaconda3\lib\site-packages\py4j\protocol.py", line 332, in get_return_value
    format(target_id, ".", name, value))

Py4JError: An error occurred while calling o102.__getstate__. Trace:
py4j.Py4JException: Method __getstate__([]) does not exist
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
    at py4j.Gateway.invoke(Gateway.java:274)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Unknown Source)

I am not getting what this error is about? and how to fix it?

$\endgroup$
2
  • $\begingroup$ Try changing sc.parallelize(train_df) to train_df.rdd in the second line and convert each row to LabeledPoint. Also, the third line should also use the same converted RDD, not a dataframe. $\endgroup$
    – Shaido
    Jun 9 '20 at 5:56
  • $\begingroup$ Or you can use the newer spark ML library (not mllib): spark.apache.org/docs/2.2.0/… $\endgroup$
    – Shaido
    Jun 9 '20 at 6:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.