I am working on translating large amounts of tweets using this deep-translator which uses the Google Translate API.
Initially everything was fine and tweets were translated with no problems whatsoever but I recently encountered an issue.
The issue with the data currently is that when there are non-English words in a particular tweet as in the attached image, Google Translate marks it as English and does not translate them. In the API I have set the source language to auto-detect the words then translate them to English.
The only workaround I have come up with to solve this is to turn the tweet into chunks and perform batch translations on them.
"NEW YOUTUBE VIDEO OUT NOW:TOTTENHAM NEWS TRANSFER WINDOW UPDATE 손흥민 Son Award Link to Premier League Defen... TOTTENHAM NEWS TRANSFER WINDOW UPDATE Carabao Cup Win Final. 손흥민 Son Contract"
Output after chunking the string into batches with a maximum of 5 words:
['NEW YOUTUBE VIDEO OUT NOW:TOTTENHAM', 'NEWS TRANSFER WINDOW UPDATE 손흥민', 'Son Award Link to Premier', 'League Defen... TOTTENHAM NEWS TRANSFER', 'WINDOW UPDATE Carabao Cup Win', 'Final. 손흥민 Son Contract']
The issue is to translate the batch it takes about 13 seconds whereas if I translate the entire string(even though it won’t work) it takes under a second as the batched string performs 6 API request compared to the single API request for the normal string.
The time to translate will be quite high when I batch the strings and translate them. Assuming it takes 13 seconds per tweet with 5000 tweets in the csv file I am currently working on it will take roughly 18 hours and I have csv files with significantly more tweets than this.
The chunk size of 5 is what I found to still be able to translate the words even if there is a single non-English word in the string. Anything more than that will still not translate the tweets.
Anyone have any workarounds that are faster than this?