Issue translating large amounts of tweets using Google Translate

Question

I am working on translating large amounts of tweets using this deep-translator which uses the Google Translate API.

Initially everything was fine and tweets were translated with no problems whatsoever but I recently encountered an issue.

The issue with the data currently is that when there are non-English words in a particular tweet as in the attached image, Google Translate marks it as English and does not translate them. In the API I have set the source language to auto-detect the words then translate them to English.

The only workaround I have come up with to solve this is to turn the tweet into chunks and perform batch translations on them.

Example string:

"NEW YOUTUBE VIDEO OUT NOW:TOTTENHAM NEWS TRANSFER WINDOW UPDATE 손흥민 Son Award Link to Premier League Defen... TOTTENHAM NEWS TRANSFER WINDOW UPDATE Carabao Cup Win Final. 손흥민 Son Contract"

Output after chunking the string into batches with a maximum of 5 words:

['NEW YOUTUBE VIDEO OUT NOW:TOTTENHAM', 'NEWS TRANSFER WINDOW UPDATE 손흥민', 'Son Award Link to Premier', 'League Defen... TOTTENHAM NEWS TRANSFER', 'WINDOW UPDATE Carabao Cup Win', 'Final. 손흥민 Son Contract']

The issue is to translate the batch it takes about 13 seconds whereas if I translate the entire string(even though it won’t work) it takes under a second as the batched string performs 6 API request compared to the single API request for the normal string.

The time to translate will be quite high when I batch the strings and translate them. Assuming it takes 13 seconds per tweet with 5000 tweets in the csv file I am currently working on it will take roughly 18 hours and I have csv files with significantly more tweets than this.

The chunk size of 5 is what I found to still be able to translate the words even if there is a single non-English word in the string. Anything more than that will still not translate the tweets.

Anyone have any workarounds that are faster than this?

Answer 1

So this was my work around:

1. Convert the list of strings that need to be translated into an indexed tuple with the first value being the index and the second value being the string:

s = [(0, 'NEW'), (1, 'YOUTUBE'), (2, 'VIDEO'), (3, 'OUT'), (4, 'NOW:TOTTENHAM'), (5, 'NEWS'), (6, 'TRANSFER'), (7, 'WINDOW'), (8, 'UPDATE'), (9, '손흥민'), (10, 'Son'), (11, 'Award'), (12, 'Link'), (13, 'to'), (14, 'Premier'), (15, 'League'), (16, 'Defen...'), (17, 'TOTTENHAM'), (18, 'NEWS'), (19, 'TRANSFER'), (20, 'WINDOW'), (21, 'UPDATE'), (22, 'Carabao'), (23, 'Cup'), (24, 'Win'), (25, 'Final.'), (26, '손흥민'), (27, 'Son'), (28, 'Contract')]

2. Next as the words that were causing issues when translating were ASCII I extracted them using this code:

(index, item) for (index, item) in s if not item.isascii()]
# result
[(9, '손흥민'), (26, '손흥민')]

3. Then I ran the translations on these words instead of the entire string or the chunks of strings.
4. After translating I then added them back in the list of tuples
5. Convert the list of tuples to a list of strings and concatenate them into a single string

This significantly reduced the time to translate the data.