Alternatives to Logistic Regression

Question

I have age, gender, height, weight and some other similar parameters of 15000 subjects. I also have one column showing if they had a medical condition (present in about 20% subjects). I now want to analyze the factors which are most contributing to the medical condition. The usual test for this is Logistic regression analysis. However, that assumes that the relationships are linear. What other machine learning tests can I apply for this purpose with this kind of data? Thanks for your insight.

Answer 1

If you choose your alternative to Tree based models, then you really have an upper edge here as compared to all other linear/logistic Regressions etc..

People generally useCo-relation, Co-variance and heat maps.. etc as a process which often generates tables of distances with even more numbers than the original data, but making Dendrograms in fact simplifies our understanding of the data.

Distances between objects can be visualized in many simple and evocative ways, one of them is Hierarchial Clustering..

What is Hierarchical Clustering?

Hierarchical clustering is where you build a cluster tree (a dendrogram) to represent data, where each group (or “node”) links to two or more successor groups. The groups are nested and organized as a tree, which ideally ends up as a meaningful classification scheme.

So now, What is a Dendrogram?

A dendrogram is a type of tree diagram showing hierarchical clustering — relationships between similar sets of data. They are frequently used in biology to show clustering between genes or samples, but they can represent any type of grouped data.

The columns under the same split at the leaves are somewhat having relationships between them or have similar attributes.., that is what we try to Explore and deepen out understanding about in order to cut down redundant Features....

Sample Dendrogram looks like this one...

So Let's try to interpret the Diagram Now,

Removing redundant features

One thing that makes it harder to interpret a variable is that there seem to be some variables with very similar meanings(redundant features...)

Let's try removing some of these related features to see if the model can be simplified without impacting the accuracy.

def get_oob(df):
    m = RandomForestRegressor(n_estimators=30, min_samples_leaf=5, max_features=0.6, n_jobs=-1, oob_score=True)
    x, _ = split_vals(df, n_trn)
    m.fit(x, y_train)
    return m.oob_score_

Here's our baseline.

get_oob(df_keep)
0.88999425494301454

Now we try removing each variable one at a time....

for c in ('saleYear', 'saleElapsed', 'fiModelDesc', 'fiBaseModel', 'Grouser_Tracks', 'Coupler_System'):
    print(c, get_oob(df_keep.drop(c, axis=1)))

Output(s)

saleYear 0.889037446375
saleElapsed 0.886210803445
fiModelDesc 0.888540591321
fiBaseModel 0.88893958239
Grouser_Tracks 0.890385236272
Coupler_System 0.889601052658

It looks like we can try one from each group for removal. Let's see what that does.

to_drop = ['saleYear', 'fiBaseModel', 'Grouser_Tracks']

Looks good even after dropping some of the columns....

References -:

• Wiki Link

• Notebook Link - fast.ai(Jeremy is Just Awesome..)

• Blog Link

Edit - (just to make the answer complete)

We also have something known as Partial Dependencies while using RF's which is also a very helpful insight to explore even further...

Answer 2

Any classifier that naturally provides feature importance can be applied here. For example, take a look at Decision Trees and Random Forrest.

For other algorithms that do not provide such information (eg KNN and Neural Nets), you may take a (forward/backward/bidirectional) feature selection. For example, chose a subset of features, train a model and evaluate it with any preferred metrics (better to use bootstrapping or cross-validation). Choose another subset if features and try again. After several trials, you can find the subset with the best prediction power. This is the set of the mist contributing features.

Note that different techniques/measures may give you different results.

Answer 3

Besides the obvious ideas regarding machine learning methods such as tree based approaches (e.g. xgboost), I would note that logistic regression is not limited to linear effects. For example, spline based methods can be applied quite easily (see e.g. Frank Harrell's "Regression Modeling Strategies - With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis" for more details). There are also generalized additive models, that may be yet another option. In such models it may be slightly easier to interpret the contribution of different variables than in tree based methods, where this can be a bit harder.

Answer 4

I now want to analyze the factors which are most contributing to the medical condition

If this is your purpose I suggest you plotting covariance matrix for input features. By doing so you will be able to see the correlation of your features together. That may help you diminish redundant correlated features 1. Moreover, you can plot the correlation of your outcomes and each feature to see their relationship. That helps you figure out which features are correlated or uncorrelated to the outcomes.

PCA helps you find major components are needed to describe the phenomena 2. Neural nets are useful for making a model that can predict for labeled data-sets. For your case, I've not seen any paper but there might be.