Determining usage/context of terms/acronyms/abbreviations in python

Question

Background: I work for a major airline in the United States and have been given a pretty significant text mining and machine learning task. Since there are several unique terms, acronyms, and abbreviations in aviation, I have written a program that reads through the documents. The program is using 4 groups of terms; WORDS, STOPS, AIRPORTS, and SYNONYMS. WORDS, STOPS, and AIRPORTS are all sets where the first two were initialized from NLTK and the last was initialized with a manually built list of airports that my employer services. SYNONYMS is a nested dictionary 'trie' style structure where the branches are misspellings and the leaves are the terms that they represent. it was initialized as an empty dictionary. I have also created a command prompt UI that informs me when it finds a word that it doesn't know and allows me to correct the word or add it to one of the aforementioned groups. The documents are being tokenized by spaCy and fed into scikit-learn machine learning algorithms.

Problem: I've started noticing that some of the words that I previously accepted in WORDS or SYNONYMS are showing up with alternate meanings and could have a drastic impact on how well the algorithm operates. For example the word act can be the verb act; or it can be an abbreviation for actuator or active; or it can be an acronym for auxiliary central tank or a couple other things. At present, I'm using the UI to manually correct or ignore these occurrences, which is time consuming, cumbersome, and prone to errors.

Desired result: What I'm looking for is a method of determining whether 'lt' means left or light depending on context (lt wing vs. warning lt) with minimal manual entry. At a minimum I would like some high level ideas/discussion on how this may be accomplished in python. Optimally, I would like some example code, preferably using the same tools I already have; spaCy, NLTK, and sklearn but I am flexible and willing to learn.

Notes: I've considered a couple methods of doing this but I haven't been able to come up with any ideas strong enough to try. For example, I've considered marking the terms with the pos inside the sets by concatenating it with the word before adding it like 'rat:noun'. The issue with this is that rat:noun could be an animal or it could be an abbreviation for ram air turbine, which is also a noun.

Answer 1

After rubber ducking with a friend of mine, I decided to take a two stage approach.

First, whenever I see a word I've recognized as a multiple use word, I flag it in the system and give it the appropriate correction. Once it evaluates that part, it stores the corrected word and enough words such that there is sufficient context in a tuple... inside a list... inside a dictionary. It all looks something like use_case = {'abbr': ('five word abbr context string', 'correction').

Secondly, I've created a dictionary of scikit-learn LinearSVCs that looks like svm_dict = {'abbr': LinearSVC}.

Once these things are established and the program is run again, when it finds one of the multiple usage words it first checks use_case to see if that exact set of 5 words surround this case. If it does it swaps it out for the original correction, otherwise it tries to predict the usage with the appropriate SVC and asks the user if the prediction is correct.

I know it's not fully automated, but neither is the program in which it runs. I figured the occasional yes/no question to the user was easier than leaving it off the lists and rewriting it every... single... time...

Either way, I have it all in place but haven't gotten to the point where it's being used yet. So, I'll either be editing, deleting, or accepting this answer sometime next week when I've had some time to test'r out.

Answer 2

As an extension to Eric Ed Lohmar's answer, This reference suggests using a mapping between:

[short form][long form] 
[long form][short form]

with a rule set based on the number of characters in the short form. However, I prefer the SVC approach above and wonder if Naive Bayes classifiers would work well for this task too.

Also, as capitalization is important, another consideration is to attempt both short and long forms with and without stop words. For example you might have

{"MPD":"The Metropolitan Police Department","Metropolitan Police Department"}
{"TDN":"The Dental Network"}

Some related SO questions:

Question 1 Question 2