0
$\begingroup$

I am having trouble using the Tokenizer

The code is

from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences

tokenizer = Tokenizer()
tokenizer.fit_on_texts(df.descricao_despesa)



X = tokenizer.texts_to_sequences(df.descricao_despesa)


maxlen = 100

X= pad_sequences(df.descricao_despesa, padding='post', maxlen=maxlen)

I tried to cast column descricao_despesa to String but still doest work.

If i try to cast to StringType i receive

from pyspark.sql.functions import col
df = df.withColumn("descricao_despesa", col("descricao_despesa").cast('StringType'))

ParseException: u'\nDataType stringtype() is not supported.(line 1, pos 0)\n\n== SQL ==\nStringType\n^^^\n

I am using spark 2.0 and Python 2.7

$\endgroup$
2
$\begingroup$

I'm not sure that one can use Keras on Spark to process data in parallel (using multiple workers).

But if your data can fit in RAM on a single node, that you can easily create a Pandas DataFrame from a Spark DataFrame and pass that DF to Keras. You can also convert result back to Spark DF if you need.

Spark DF --> Pandas DF:

pdf = df.toPandas() # df - is a Spark DataFrame

Pandas DF --> Spark DF:

sdf = spark.createDataFrame(pdf)
| improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.