1
$\begingroup$

I am a novice in machine learning and new to NLP. I am looking for ideas on how to solve the below two problems.

I have a dataset with two columns, "Titles" and "Description". Titles column has names of clinical lab tests and description column has description about results of corresponding laboratory test( can be seen below). There many titles specific to a particular lab test.

Title                       Description
Complete blood test         RBC: Normocytic and Normochromic
COMPLETE Blood test\        Platelets: Adequate on the smear
Blood glucose               COLOUR - COLOURLESS
Complete blood picture      WBC: Total and Differential counts are within normal limits

I have only shared a small part of the data frame.

Problem 1: I have manually grouped the similar looking titles. For example, I have grouped the titles in the above data frame as "Blood Test". Is there a way to use NLP technique to group similar looking titles ( As shown below).

Problem 2: Based on the description, i have manually labeled a particular outcome for a lab test as normal or abnormal. Again i am looking for a way to do this without having to manually label the outcome (As shown below).

 Title                       Description                           Outcome
Blood Test                   RBC: Normocytic and Normochromic       Normal
Blood Test                   Platelets: Adequate on the smear       Normal
Blood Test                   COLOUR - COLOURLESS                    Normal
Blood Test                   WBC: Total and Differential counts     Normal

Any suggestions or resources to help me get started would be appreciated.

$\endgroup$
0
$\begingroup$

Take a supervised approach:

For each group "Blood test", "Stool test" etc,

  • Take a subset of your rows as a training set, say 200 rows.
  • Create a new numeric or logical column, "IsBloodTest" or similar for this new subset of your dataset

For each row in these 200

  • Classify them manually: if it is a blood test, assign 1, if not assign 0 to "IsBloodTest"
  • Split the "Description" column into word vectors. This creates a Document-Term matrix with only 1 or 0 values in the cells (Word is present/not present)
  • Classify the rows of the sparse new matrix (omitting your own newly manually created attribute) with the Multinomial NaiveBayes algorithm. This assigns another column with "0/1" prediction, say "PredictedBloodTest" to your training set. With these 0/1 values in the two new columns you can build a confusion matrix.

Perform Cross validation experiments to check your confusion matrix, or use an extra validation set. Label more rows manually if necessary.

If the classification result is good enough, use it for the remaining 1000s of rows from your data set.

This procedure is simple, but it might get you results faster than learning more sophisticated approaches first.

$\endgroup$
  • $\begingroup$ Could you give any suggestion on the second problem? $\endgroup$ – Karthik Shanmukha Dec 20 '17 at 11:08
  • $\begingroup$ I think the second problem needs substantial domain knowledge. Do you have more attributes at your disposal? Or only the general description. I don't understand the second question. $\endgroup$ – knb Dec 20 '17 at 11:27
  • $\begingroup$ Title column has other titles besides blood test, like xray etc. There are about 30 titles. How can i group them all? $\endgroup$ – Karthik Shanmukha Dec 21 '17 at 6:53
  • $\begingroup$ Sorry, I don't know. maybe "Topic-Modeling" is an alternative, or "Multiclass-Classification" in a "One-versus-All appoach", as the categories are known in advance. Seems like you need to consult a true expert in this, and a huge training data set. - Or proceed by classifiying manually using the method sketched above, but this may be a lot of work. Besides, thanks for accepting my answer. $\endgroup$ – knb Dec 21 '17 at 8:10
0
$\begingroup$

Problem 1: Approach 1 : Do you have with you the possible types of tests that can be conducted? If yes, then as suggested above - assign each row to one of the test type. Then build the classifier model based on this. This model can be used to predict the test type for new data.

Approach 1 : If you do not have the information about what are the possible test types and you want the machine to find the pattern, then go for clustering approach , where similar titles will be grouped together, and you will be getting multiple such groups.

Problem 2: Clearly you have only two output classes , its a classification problem.You need labelled data initially to build the model. You will have to (again) initially label data based on your domain knowledge, and then feed it to the classification algorithm.

If using R - you can use the tm package , if python then nltk should help you.The choice of which algorithm to use depends on mulitple factors.Do let me know if you need more info.

$\endgroup$
  • $\begingroup$ thank you for your suggestions. With respect to problem 1, i would want to build a classifier using existing data and make prediction on new data, as more data is expected to come in. I have the possible names of the tests, which are around 30. I could group a few test names together and bring it down to 10. i am familiar with R but i would want to implement it in python. If you could share any resources regarding the same, it would be great! $\endgroup$ – Karthik Shanmukha Dec 26 '17 at 4:12
  • $\begingroup$ i have grouped the tests names as following: Body fluid analysis Diagnostic imaging tests, Cytology tests, Organ function tests, Patient related, Doctors Advice, & Miscellaneous. $\endgroup$ – Karthik Shanmukha Dec 26 '17 at 4:36
  • $\begingroup$ I could probably do a multi-label classification. $\endgroup$ – Karthik Shanmukha Dec 26 '17 at 4:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.