Predicting next medical condition from past conditions in claims data

Question

I am currently working with a large set of health insurance claims data that includes some laboratory and pharmacy claims. The most consistent information in the data set, however, is made up of diagnosis (ICD-9CM) and procedure codes (CPT, HCSPCS, ICD-9CM).

My goals are to:

1. Identify the most influential precursor conditions (comorbidities) for a medical condition like chronic kidney disease;
2. Identify the likelihood (or probability) that a patient will develop a medical condition based on the conditions they have had in the past;
3. Do the same as 1 and 2, but with procedures and/or diagnoses.
4. Preferably, the results would be interpretable by a doctor

I have looked at things like the Heritage Health Prize Milestone papers and have learned a lot from them, but they are focused on predicting hospitalizations.

So here are my questions: What methods do you think work well for problems like this? And, what resources would be most useful for learning about data science applications and methods relevant to healthcare and clinical medicine?

EDIT #2 to add plaintext table:

CKD is the target condition, "chronic kidney disease", ".any" denotes that they have acquired that condition at any time, ".isbefore.ckd" means they had that condition before their first diagnosis of CKD. The other abbreviations correspond with other conditions identified by ICD-9CM code groupings. This grouping occurs in SQL during the import process. Each variable, with the exception of patient_age, is binary.

Answer 1

I've never worked with medical data, but from general reasoning I'd say that relations between variables in healthcare are pretty complicated. Different models, such as random forests, regression, etc. could capture only part of relations and ignore others. In such circumstances it makes sense to use general statistical exploration and modelling.

For example, the very first thing I would do is finding out correlations between possible precursor conditions and diagnoses. E.g. in what percent of cases chronic kidney disease was preceded by long flu? If it is high, it doesn't always mean causality, but gives pretty good food for thought and helps to better understand relations between different conditions.

Another important step is data visualisation. Does CKD happens in males more often than in females? What about their place of residence? What is distribution of CKD cases by age? It's hard to grasp large dataset as a set of numbers, plotting them out makes it much easier.

When you have an idea of what's going on, perform hypothesis testing to check your assumption. If you reject null hypothesis (basic assumption) in favour of alternative one, congratulations, you've made "something real".

Finally, when you have a good understanding of your data, try to create complete model. It may be something general like PGM (e.g. manually-crafted Bayesian network), or something more specific like linear regression or SVM, or anything. But in any way you will already know how this model corresponds to your data and how you can measure its efficiency.

---

As a good starting resource for learning statistical approach I would recommend Intro to Statistics course by Sebastian Thrun. While it's pretty basic and doesn't include advanced topics, it describes most important concepts and gives systematic understanding of probability theory and statistics.

Answer 2

While I am not a data scientist, I am an epidemiologist working in a clinical setting. Your research question did not specify a time period (ie odds of developing CKD in 1 year, 10 years, lifetime?).

Generally, I would go through a number of steps before even thinking about modeling (univariate analysis, bivariate analysis, colinearity checks, etc). However, the most commonly used method for trying to predict a binary event (using continuous OR binary variables) is logistic regression. If you wanted to look at CKD as a lab value (urine albumin, eGFR) you would use linear regression (continuous outcome).

While the methods used should be informed by your data and questions, clinicians are used to seeing odds ratios and risk ratios as these the most commonly reported measures of association in medical journals such as NEJM and JAMA.

If you are working on this problem from a human health perspective (as opposed to Business Intelligence) this Steyerberg's Clinical Prediction Models is an excellent resource.

Answer 3

"Identify the most influential precursor conditions (comorbidities) for a medical condition like chronic kidney disease"

I'm not sure that it's possible to ID the most influential conditions; I think it will depend on what model you're using. Just yesterday I fit a random forest and a boosted regression tree to the same data, and the order and relative importance each model gave for the variables were quite different.