I am having trouble using the Tokenizer

The code is

from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences

tokenizer = Tokenizer()

X = tokenizer.texts_to_sequences(df.descricao_despesa)

maxlen = 100

X= pad_sequences(df.descricao_despesa, padding='post', maxlen=maxlen)

I tried to cast column descricao_despesa to String but still doest work.

If i try to cast to StringType i receive

from pyspark.sql.functions import col
df = df.withColumn("descricao_despesa", col("descricao_despesa").cast('StringType'))

ParseException: u'\nDataType stringtype() is not supported.(line 1, pos 0)\n\n== SQL ==\nStringType\n^^^\n

I am using spark 2.0 and Python 2.7


I'm not sure that one can use Keras on Spark to process data in parallel (using multiple workers).

But if your data can fit in RAM on a single node, that you can easily create a Pandas DataFrame from a Spark DataFrame and pass that DF to Keras. You can also convert result back to Spark DF if you need.

Spark DF --> Pandas DF:

pdf = df.toPandas() # df - is a Spark DataFrame

Pandas DF --> Spark DF:

sdf = spark.createDataFrame(pdf)

Not the answer you're looking for? Browse other questions tagged or ask your own question.