Predicting which drug is most appropriate for which patient returns an accuracy of almost 0

Question

I have a dataframe that looks like this:

data = {'age': [54, 21, 7, 18],
        'sex': [0, 1, 1, 0],
        'disease_type': ['A', 'B', 'A', 'F'],
        'change_in_pain': [-0.54, -0.89, 0.07, -0.01],
        'drug': ['drug_1', 'drug_7', 'drug_1', 'drug_89'],
        }
df = pd.DataFrame(data)

=>

   age  sex disease_type  change_in_pain     drug
0   54    0            A           -0.54   drug_1
1   21    1            B           -0.89   drug_7
2    7    1            A            0.07   drug_1
3   18    0            F           -0.01  drug_19
...

The real df has > 10000 rows (=patients) and 34 different drugs but seemingly I cant upload a csv here for a more usable example?

I would like to train a model that predicts which drug is most effective for which patient given the patient’s age, sex, disease type and how much the pain was reduced (a more negative “change_in_pain” column is better).

In this simple example “drug_1” woud work for disease A only if the patient is older and female.

I wrote the following code but the mean accuracy is returned as almost 0 :

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
   

# shuffle
df = df.sample(frac=1.0).reset_index(drop=True)

X = df[['age', 'sex', 'disease_type', 'change_in_pain']]
y = df['drug']

# convert categorical variable into dummy/indicator variables.
X_OHE = pd.get_dummies(X)
y_OHE = pd.get_dummies(y)

X_train, X_test, y_train, y_test = train_test_split(X_OHE, y_OHE, test_size=0.20)

scaler = StandardScaler()
scaler.fit(X_train)

X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

knn = KNeighborsClassifier(5)
knn.fit(X_train, y_train)
score = knn.score(X_test, y_test)
print('mean accuracy: {:2.2f}'.format(score))

I also tested: RandomForestClassifier(max_depth=5, n_estimators=10, max_features=1), KNeighborsClassifier(3), DecisionTreeClassifier(max_depth=5), MLPClassifier(alpha=1, max_iter=1000) but again the mean acc is around zero.

What am I doing wrong?

EDIT:

Doing it more slowly using:

knn.fit(X_train, y_train)  # X_train: 8000x11, y_train: 8000x34
y_pred = clf.predict(X_test)  # X_test: 2000x11, y_pred: 2000x34
acc = accuracy_score(y_test, y_pred)

shows that y_pred seems to contain only zeros - but why?

Answer 1

I can't say exactly why you get null accuracy, but I have some comments that may help:

• you are mixing continous data and categorical data. You might already be aware:
  • when standardizing your data: you are although standardizing your categorical data (sex, disease_type), just be sure that it makes sense (depending on the classifier you're using)
  • depending on the classifier, mixing continous and binary data may lead to undesirable results. This post explains it in the case of KNN

Otherwise I don't see anything suspicous after a first look. Could you share more data so we can reproduce your result ?

Answer 2

You should visualize your data to see what kind of decision bound might fit. Possibly, there is no weighting of features that can predict drug type.