I am a bit new to the field of data science and could really use some help. I used a natural language dictionary to train and test an ml model using keras and tensorflow. It detects the sentiments in a string and returns 0 or 1. Now, I have a dataset containing about 200,000 rows with each row containing 1 or 2 paragraphs of text and I wrote a quick function that checks and returns the sentimental polarity of each row and I am currently using a single for loop to parse through the dataset, check the string from each row and append the polarity into an empty column. It works perfectly when I tried it on a smaller subset of 100 rows.
The above process is extremely slow and on my current setup and 200,000 rows will take maybe more than a week to process. I am using sagemaker notebooks with a c5.xlarge instance currently and cant afford to get better compute hardware. Do you guys have any advice on how to deal with this? Any help will be much appreciated!
def predictions(texter): texter = tokenizer.texts_to_sequences(texter) texter = pad_sequences(texter, maxlen = 96, dtype = 'int32', value = 0) sentiment = model.predict(texter ,batch_size = 1, verbose = 0) sentiment = [float(a)/sum(sentiment) for a in sentiment] if ((sentiment*100)>75): return '0' else: return '1' for i in range (len(df['body'])): df['bodysentiment'][i] = predictions(df['body'][i]) ```