Grouping of similar looking text

Question

I have a data frame which has two columns, "Title" and "Description". The title column has a bunch of titles related to clinical lab tests. Unfortunately, most of the titles are a repeat of the same test but, due to minor changes in the titles, titles are shown as unique.

values = [('Complete blood picture', 'AB'), ('Complete BLOOD test', 'AB'), ('blood glucose', 'AB'), ('COMplete blood Profile', 'AB')]

labels = ['title', 'description']
import pandas as pd
labtest = pd.DataFrame.from_records(values, columns = labels) # Create data frame

This is how the data frame looks like. [The actual dataset has many such titles, this is only for the purpose of this question]

Title                       Description
Complete blood test         AB
COMPLETE Blood test\        AB
Blood glucose               AB
Complete blood picture      AB

And this is what I would the like the data frame to look like:

Title                       Description
Blood test                   AB
Blood test                   AB
Blood test                   AB
Blood test                   AB

I would like to search for the word "blood" in each title and if its true, then change the whole title with "Blood test". Is there a way to do this?

Answer 1

One possible solution is the following:

import re
import pandas as pd

pattern = re.compile('blood', re.IGNORECASE)

def change(text):
    if pattern.findall(text):
        return 'Blood test'
    else:
        return text

values = [('Complete blood picture', 'AB'), ('Complete BLOOD test', 'AB'), ('blood glucose', 'AB'), ('COMplete blood Profile', 'AB')]
labels = ['title', 'description']

# Create data frame
labtest = pd.DataFrame.from_records(values, columns=labels)
labtest['title'] = labtest['title'].apply(change)

print labtest

The output was:

        title description
0  Blood test          AB
1  Blood test          AB
2  Blood test          AB
3  Blood test          AB

The first line import the regex (regular expression) module of Python. The line:

pattern = re.compile('blood', re.IGNORECASE)

creates a regex that finds the word blood ignoring case. The function change, replace the input text with 'Blood test' in case the string 'blood' was found. Finally you used the apply method from pandas DataFrame to transform the column. Finally the apply method, as the name suggests, 'applies' the function change to every value in the 'title' column.

More info on regular expressions using Python and pandas apply method can be found here and here. If you want to know more about text processing in Python I would recommend you take a look at the pointers in this question.

Answer 2

Pandas can directly do that string compare, then use the compare result to lookup appropriate rows so that they can be set. This can be done with a single expression as:

Code:

labtest['title'][labtest['title'].str.contains('blood', case=False)] = 'Blood test'

How does this work?

From the inside out we have:

1. Select the 'title' column as a string vector:

   labtest['title'].str
   

2. Convert the string vector to a boolean vector:

   labtest['title'].str.contains('blood', case=False)
   

3. Use the boolean vector to select specific rows in the title column:

   labtest['title'][labtest['title'].str.contains('blood', case=False)] 
   

4. Assign those cells to the desired new value

   labtest['title'][labtest['title'].str.contains('blood', case=False)] = 'Blood test'
   

Test Code:

values = [
    ('Complete blood picture', 'AB'),
    ('Complete BLOOD test', 'AB'),
    ('blood glucose', 'AB'),
    ('COMplete blood Profile', 'AB'),
    ('bloud glucose', 'AB'),
]
labels = ['title', 'description']

# Create data frame
labtest = pd.DataFrame.from_records(values, columns=labels)

labtest['title'][labtest['title'].str.contains('blood', case=False)] = 'Blood test'

print(labtest)

Test Results:

           title description
0     Blood test          AB
1     Blood test          AB
2     Blood test          AB
3     Blood test          AB
4  bloud glucose          AB

Answer 3

Another solution:

new_values=[]
for tup in values:
    if tup[0].lower().find('blood')>=0:
        new_values.append(['Blood test',tup[1]])
    else: new_values.append([ tup[0],tup[1] ])

This basically takes your values list and creates an new_values list with the replaced text. See below:

values =     [('Complete blood picture', 'AB'), ('Complete BLOOD test', 'AB'), ('blood glucose', 'AB'), ('COMplete blood Profile', 'AB')]
new_values = [['Blood test', 'AB'], ['Blood test', 'AB'], ['Blood test', 'AB'], ['Blood test', 'AB']]

So now you can use new_values in your dataframe (pd.DataFrame.from_records(new_values, columns = labels)) or use it to replace values.