1
$\begingroup$

I am a bit new to the field of data science and could really use some help. I used a natural language dictionary to train and test an ml model using keras and tensorflow. It detects the sentiments in a string and returns 0 or 1. Now, I have a dataset containing about 200,000 rows with each row containing 1 or 2 paragraphs of text and I wrote a quick function that checks and returns the sentimental polarity of each row and I am currently using a single for loop to parse through the dataset, check the string from each row and append the polarity into an empty column. It works perfectly when I tried it on a smaller subset of 100 rows.

The above process is extremely slow and on my current setup and 200,000 rows will take maybe more than a week to process. I am using sagemaker notebooks with a c5.xlarge instance currently and cant afford to get better compute hardware. Do you guys have any advice on how to deal with this? Any help will be much appreciated!

def predictions(texter):
  texter = tokenizer.texts_to_sequences(texter)
  texter = pad_sequences(texter, maxlen = 96, dtype = 'int32', value = 0)
  sentiment = model.predict(texter ,batch_size = 1, verbose = 0)[0]
  sentiment = [float(a)/sum(sentiment) for a in sentiment]
  if ((sentiment[0]*100)>75):
    return '0'
  else:
    return '1'

for i in range (len(df['body'])):
  df['bodysentiment'][i] = predictions(df['body'][i])
```
Improve this question
$\endgroup$
3
  • $\begingroup$ Have you tried using .apply() instead of manually looping over the rows? Manually looping in pandas is the slowest way of applying a function to every row. In addition, can the texts_to_sequences and pad_sequences functions be applied to all rows at once? $\endgroup$
    – Oxbowerce
    Commented Jun 17, 2021 at 12:02
  • $\begingroup$ hi, thanks for the suggestions. I tried apply using this df['bodysentiment'] = df.apply(lambda row: predictions(df['body']), axis=1) but it didnt make much difference on the 100 rows set, a for loop timed at 45 seconds, and .apply took 42 seconds. Although it helps, i want the process to be way more quicker. $\endgroup$
    – Glad
    Commented Jun 17, 2021 at 12:23
  • $\begingroup$ As Nicolas points out, start by looking which of lines in your function are taking the most time to see where to focus your efforts. $\endgroup$
    – Oxbowerce
    Commented Jun 17, 2021 at 13:03

1 Answer 1

Reset to default
1
$\begingroup$

Here are a few suggestions:

  • Detect the lines of code that last longer than the other with the Python's time function.
  • Generally speaking, PC processors have 4 cores and you can benefit from each of them if you apply multiprocessing with pandas thanks to the multiprocessing library.
  • Pypy is a very good library to run code very fast, even faster than Cython (https://www.pypy.org/)
  • .iterrows() or .loc could perform better (for loops are too slow).
  • Google collab uses plenty of powerful servers that could process lot of data in minutes instead of days in many cases.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.