I have a dataframe that looks like this:

data = {'age': [54, 21, 7, 18],
        'sex': [0, 1, 1, 0],
        'disease_type': ['A', 'B', 'A', 'F'],
        'change_in_pain': [-0.54, -0.89, 0.07, -0.01],
        'drug': ['drug_1', 'drug_7', 'drug_1', 'drug_89'],
df = pd.DataFrame(data)


   age  sex disease_type  change_in_pain     drug
0   54    0            A           -0.54   drug_1
1   21    1            B           -0.89   drug_7
2    7    1            A            0.07   drug_1
3   18    0            F           -0.01  drug_19

The real df has > 10000 rows (=patients) and 34 different drugs but seemingly I cant upload a csv here for a more usable example?

I would like to train a model that predicts which drug is most effective for which patient given the patient’s age, sex, disease type and how much the pain was reduced (a more negative “change_in_pain” column is better).

In this simple example “drug_1” woud work for disease A only if the patient is older and female.

I wrote the following code but the mean accuracy is returned as almost 0 :

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

# shuffle
df = df.sample(frac=1.0).reset_index(drop=True)

X = df[['age', 'sex', 'disease_type', 'change_in_pain']]
y = df['drug']

# convert categorical variable into dummy/indicator variables.
X_OHE = pd.get_dummies(X)
y_OHE = pd.get_dummies(y)

X_train, X_test, y_train, y_test = train_test_split(X_OHE, y_OHE, test_size=0.20)

scaler = StandardScaler()

X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

knn = KNeighborsClassifier(5)
knn.fit(X_train, y_train)
score = knn.score(X_test, y_test)
print('mean accuracy: {:2.2f}'.format(score))

I also tested: RandomForestClassifier(max_depth=5, n_estimators=10, max_features=1), KNeighborsClassifier(3), DecisionTreeClassifier(max_depth=5), MLPClassifier(alpha=1, max_iter=1000) but again the mean acc is around zero.

What am I doing wrong?


Doing it more slowly using:

knn.fit(X_train, y_train)  # X_train: 8000x11, y_train: 8000x34
y_pred = clf.predict(X_test)  # X_test: 2000x11, y_pred: 2000x34
acc = accuracy_score(y_test, y_pred)

shows that y_pred seems to contain only zeros - but why?

  • 1
    $\begingroup$ You predict "drug" but you say you want to see "which drug is most effective". Why don't you look at change_in_pain = drug + ... ? Isn't this what means "effectiveness" here? $\endgroup$ – Peter Oct 15 '20 at 17:32
  • $\begingroup$ ... good point! I mean i want to predict which drug to use but I could re-formulate the question like this: ['age', 'sex', 'disease_type', 'drug'] => 'change_in_pain' and then predict every drug for a patient and take the drug with the largest reduction in pain $\endgroup$ – lordy Oct 16 '20 at 11:07
  • $\begingroup$ My take is that this is a case for "causal modeling", where you estimate the marginal effect of a drug on pain contolling for age, sex, disease_type etc. This is where I would start. Probably you would need to run different regressions for each disease_type. Depends on the data (and the possible interactions in it which you would need to model somehow). $\endgroup$ – Peter Oct 16 '20 at 11:13

I can't say exactly why you get null accuracy, but I have some comments that may help:

  • you are mixing continous data and categorical data. You might already be aware:
    • when standardizing your data: you are although standardizing your categorical data (sex, disease_type), just be sure that it makes sense (depending on the classifier you're using)
    • depending on the classifier, mixing continous and binary data may lead to undesirable results. This post explains it in the case of KNN

Otherwise I don't see anything suspicous after a first look. Could you share more data so we can reproduce your result ?

  • $\begingroup$ I could share a csv ... but how can I post it here? $\endgroup$ – lordy Oct 16 '20 at 11:02
  • $\begingroup$ You can't (see this post). You could try to extract a small amount of representative data that reproduce your issue (that would be the best) or share the full data using a free web platform. The web platform should really be the last try. $\endgroup$ – etiennedm Oct 16 '20 at 14:10

You should visualize your data to see what kind of decision bound might fit. Possibly, there is no weighting of features that can predict drug type.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.