Text mining for text matching

Question

I m new in text analysis and need your advice to help medical students to write properly and correctly. The students describe sicknesses as they observe them; however, they must use an "official sickness description". I have the data collected from the students and the correct data. Let me explain:

I have a csv table1 containing 300K rows / one column : each row is the description of health condition written by students ( they can be redundant).

I have an other table2 containing almost 200K rows and 3 columns Column 1: the official name of the condition ( the correct one the student should use) Column 2: a code ( number) Column 3: the translation of "the official name of the condition" in another language.

The idea is to start from table1 and match each row with N official names (with their codes and translation). Let's say 5 official names ranked by distance computation.

I am confused: Should I go for a recommendation engine or a search/rank algorithms?

What are the steps you can identify to accomplish this task?

PS: the final output might be an API where student start to write the description of the sickness and obtain a list of choices/recommendations where they can pick one. I hope I was clear ! Thanks

Answer 1

FuzzyWuzzy would be a good python library for this purpose. A code example from the GitHub readme:

>>> choices = ["Atlanta Falcons", "New York Jets", "New York Giants", "Dallas Cowboys"]
>>> process.extract("new york jets", choices, limit=2)
    [('New York Jets', 100), ('New York Giants', 78)]
>>> process.extractOne("cowboys", choices)
    ("Dallas Cowboys", 90)

I imagine you would like to use the extract function with limit 5 with choices being set equal to all the 'official names' from your table2 csv file.

Details on installation and use are at the link above.

Answer 2

The idea is to start from table1 and match each row with N official names (with their codes and translation). Let's say 5 official names ranked by distance computation.

I would recommend against using an ML classifier because you're likely to exclude some of the 200k official names if only 5 options are given back.

There are many fuzzy text matching algorithms to match your rows to an official name. FuzzyWuzzy's and several other algorithms are based on the Levenshtein distance. You can use a for-loop to go through the 200k official names. Depending on how much text there is this might take a while. Sort the list and slice/pop values that have to low scores, and you will have the top 5 best matching official names for a row. I guess you want to rank the frequency of each official name and return the highest ranked as a first result.

Because the health descriptions given by the students can be redundant, you might want run a Zipf ranking algorithm to filter more common words.

This is answer is far from complete. Comments are welcome.

Answer 3

Thank you all I will definitely test the FuzzyWuzzy Lib. Meanwhile please take a look on the code I have made yesterday ( it can be enhanced , Go ahead if you can make it better : –

#define function to read the CSV files
def GetDocument(link):
    dataframe = pd.read_csv(link, encoding = "ISO-8859-1", sep = ';')
    return dataframe
#define function convert text/CSV data to lists 
def ColumnToList(column):
    return column.tolist()
#read document csv 
document1 = GetDocument('C:/Users/smegrhi/Desktop/DOC1.csv')
document2 = GetDocument('C:/Users/smegrhi/Desktop/DOC2.csv')
#get columns into lists
list1 = ColumnToList(document1['DIA_LBL'])
list2 = ColumnToList(document2['ICD10_LBL_FR'])

#Data pre-processing

def ProcessText(sentence):

    #Convert to lower case
    sentence = sentence.lower()
    #Convert www.* or https?://* to URL
    sentence = re.sub('((www\.[^\s]+)|(https?://[^\s]+))','',sentence)
#Convert 
    sentence = re.sub('@[^\s]+','',sentence)
#Remove additional white spaces
    sentence = re.sub('[\s]+', ' ', sentence)
#Replace 
    sentence = re.sub(r'#([^\s]+)', r'\1', sentence)
    return sentence


#execute
for i,text in enumerate(list1):
    res = ProcessText(text)
    list1[i] = res

#vectorize the data
WORD = re.compile(r'\w+')
#function vectorize data
def text_to_vector(text):
    words = WORD.findall(text)
    return Counter(words)

#calculate similarity
def get_cosine(vec1, vec2):
    intersection = set(vec1.keys()) & set(vec2.keys())
    numerator = sum([vec1[x] * vec2[x] for x in intersection])

    sum1 = sum([vec1[x]**2 for x in vec1.keys()])
    sum2 = sum([vec2[x]**2 for x in vec2.keys()])
    denominator = math.sqrt(sum1) * math.sqrt(sum2)

    if not denominator:
        return 0.0
    else:
        return float(numerator) / denominator

# compare sentence from list 1 with list 2 : Choose Indication from Data/DOC1 

text1 = list1[1]
#text1 = shigellose à shigella dysenteriae
vector1 = text_to_vector(text1)
listCosine = []
for i,text in enumerate(list2):
    vector2 = text_to_vector(text)
    cosine = get_cosine(vector1, vector2)
    listCosine.append(cosine)

The results are not that bad neither good ... Your suggestions ?