Text classification problem using Python or R

Question

I am a novice in machine learning and new to NLP. I am looking for ideas on how to solve the below two problems.

I have a dataset with two columns, "Titles" and "Description". Titles column has names of clinical lab tests and description column has description about results of corresponding laboratory test( can be seen below). There many titles specific to a particular lab test.

Title                       Description
Complete blood test         RBC: Normocytic and Normochromic
COMPLETE Blood test\        Platelets: Adequate on the smear
Blood glucose               COLOUR - COLOURLESS
Complete blood picture      WBC: Total and Differential counts are within normal limits

I have only shared a small part of the data frame.

Problem 1: I have manually grouped the similar looking titles. For example, I have grouped the titles in the above data frame as "Blood Test". Is there a way to use NLP technique to group similar looking titles ( As shown below).

Problem 2: Based on the description, i have manually labeled a particular outcome for a lab test as normal or abnormal. Again i am looking for a way to do this without having to manually label the outcome (As shown below).

 Title                       Description                           Outcome
Blood Test                   RBC: Normocytic and Normochromic       Normal
Blood Test                   Platelets: Adequate on the smear       Normal
Blood Test                   COLOUR - COLOURLESS                    Normal
Blood Test                   WBC: Total and Differential counts     Normal

Any suggestions or resources to help me get started would be appreciated.

Answer 1

Take a supervised approach:

For each group "Blood test", "Stool test" etc,

• Take a subset of your rows as a training set, say 200 rows.
• Create a new numeric or logical column, "IsBloodTest" or similar for this new subset of your dataset

For each row in these 200

• Classify them manually: if it is a blood test, assign 1, if not assign 0 to "IsBloodTest"
• Split the "Description" column into word vectors. This creates a Document-Term matrix with only 1 or 0 values in the cells (Word is present/not present)
• Classify the rows of the sparse new matrix (omitting your own newly manually created attribute) with the Multinomial NaiveBayes algorithm. This assigns another column with "0/1" prediction, say "PredictedBloodTest" to your training set. With these 0/1 values in the two new columns you can build a confusion matrix.

Perform Cross validation experiments to check your confusion matrix, or use an extra validation set. Label more rows manually if necessary.

If the classification result is good enough, use it for the remaining 1000s of rows from your data set.

This procedure is simple, but it might get you results faster than learning more sophisticated approaches first.

Answer 2

Problem 1: Approach 1 : Do you have with you the possible types of tests that can be conducted? If yes, then as suggested above - assign each row to one of the test type. Then build the classifier model based on this. This model can be used to predict the test type for new data.

Approach 1 : If you do not have the information about what are the possible test types and you want the machine to find the pattern, then go for clustering approach , where similar titles will be grouped together, and you will be getting multiple such groups.

Problem 2: Clearly you have only two output classes , its a classification problem.You need labelled data initially to build the model. You will have to (again) initially label data based on your domain knowledge, and then feed it to the classification algorithm.

If using R - you can use the tm package , if python then nltk should help you.The choice of which algorithm to use depends on mulitple factors.Do let me know if you need more info.