How to replace short words into full words from tweets using python

Question

I am doing sentiment analysis on tweets. Most of the tweets contains short words and i want to replace them as original/full word.

Suppose that tweet is:

I was wid Ali.

I want to convert:

wid -> with

Similarly

wud -> would

i have 6000 tweets in which there are lots of short words. How i can replace them ? is there any library available in python for this task? or any dictionary of shorts words available online?

i read answer of Replace appostrophe/short words in python Question but it provides dictionary of appostrophe only.

Currently i am using NLTK but this task is not possible with NLTK.

Answer 1

I have worked on lexicon level token normalization on twitter data, which seems to be very similar to what you are trying to do. In our task, we were not only normalizing short word, but also "welllll"->"well", "lysm"->"love you so much", "2"-> to, etc. Refer to the task description here. At that time we did not find a good library directly for this job.

In our approach, we resort to buidling a lookup table to replace suspicious words. The lookup table is compiled from the following resources:

Huge List of Texting and Online Chat Abbreviations

Twitter Dictionary: A Guide to Understanding Twitter Lingo

Lexical normalisation of short text messages: makn sens a #twitter

Insertion, Deletion, or Substitution? Normalizing Text Messages without Pre-categorization nor Supervision

The latter two links are two relevant papers. They both include a dataset that we used in our lookup table.

The suspicious words that need to be replaced can be simply identified by whether they are OOV, or using a more complicated machine learning approach to determine based on context.