I have been trying to fix this for 2 months now with no luck. I am doing some medical research for my study. I have a dataset that has patients diagnosis based on medical reports (Features.csv) and each patient based on that medical report has a list of diseases (Results.csv)
Here is a sample of each file.
Features.csv
filename code frequency
1006 53438000 2
1006 54706004 10
1006 65801008 1
1006 66842004 10
1006 70901006 11
1006 71388002 1
1006 71651007 1
1006 71960002 2
1006 73761001 2
1006 74016001 1
1006 77477000 1
1007 105011006 1
1007 34896006 1
1007 363680008 2
1007 399208008 2
1007 52765003 1
1007 57485005 1
1007 71388002 3
1007 73632009 1
1007 767002 1
1007 86273004 2
1008 34227000 1
1008 363679005 1
1008 42525009 1
1008 67166004 1
1008 71388002 1
1008 90205004 1
1009 104866001 1
1009 113011001 1
1009 113063008 1
1009 118635009 2
1009 122462000 1
1009 16310003 6
1009 165581004 1
1009 168537006 2
1009 169070004 1
This is Results.csv
filename result order
1006 5990 2
1006 7802 3
1006 2762 4
1006 2738 5
1006 4589 6
1006 V4575 7
1006 27651 8
1006 56400 9
1006 4019 10
1006 V103 11
1006 2449 12
1006 2724 13
1006 56210 14
1006 2859 15
1006 5779 16
1006 5566 1
1007 1892 1
1007 1970 2
1007 496 3
1007 4280 4
1007 51881 5
1007 2859 6
1007 4019 7
1007 V1011 8
1008 4321 1
1008 41400 2
1008 4019 3
1008 2724 4
1008 71590 5
1008 V4581 6
1009 0389 1
1009 5789 2
1009 5761 3
1009 5845 4
1009 51881 5
1009 1552 6
1009 5990 7
1009 4280 8
1009 5762 9
1009 57511 10
1009 25000 11
1009 V5867 12
1009 99592 13
1009 0413 14
I did this in my Python code
1- Load dataset.
2- Pivot features to have codes as columns and frequency is the value
filename 53438000 54706004 ... 90205004
1006 2 10 0
1007 0 0 0
1008 0 0 0
1009 0 0 0
3- Pivot Results and put values in array
1006 [5990, 7802, ...]
1007 [1892, 1970, ...]
4- One hot encode results
filename 1892 1970 .... 5990 7802 ...
1006 0 0 1 1
1007 1 1 0 0
5- Split dataset into Training and Test (80/20)
6- Use LogisticRegression
7- Check Accuracy
The accuracy I get is very very bad
and when I try to tweak code I get all 1s prediction!!
What am I doing wrong here?
How can I improve accuracy?
Here are the complete files.
Features
Results
and here is my code.
import pyodbc
import pandas as pd
import numpy as np
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier
from sklearn.metrics import f1_score
from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix
RecNo = "0"
pd.set_option('display.max_colwidth', 300)
conn = pyodbc.connect('Driver={SQL Server};'
'Server=DELLG3;'
'Database=NLP2;'
'Trusted_Connection=yes;')
df_features = pd.read_sql("EXEC GetFeatures " + RecNo , conn)
df_features.shape
df_results = pd.read_sql("EXEC GetResults " + RecNo , conn)
df_features = df_features.pivot(index='filename', columns='code', values='frequency')
df_features[np.isnan(df_features)] = 0
df_results = df_results.groupby('filename')[["result"]].agg(list).reset_index()
multilabel_binarizer = MultiLabelBinarizer()
multilabel_binarizer.fit(df_results['result'])
ResultsArray = multilabel_binarizer.transform(df_results['result'])
ResultsArray = ResultsArray[:, :-1]
xtrain, xval, ytrain, yval = train_test_split(df_features, ResultsArray, test_size=0.2, random_state=9)
lr = LogisticRegression()
clf = OneVsRestClassifier(lr)
# fit model on train data
clf.fit(xtrain, ytrain)
# make predictions for validation set
y_pred = clf.predict(xval)
# evaluate performance
print(f1_score(yval, y_pred, average="micro"))
#I obtain the accuracy of this fold
ac=accuracy_score(y_pred,yval)
#I obtain the confusion matrix
cm=confusion_matrix(yval.ravel(), y_pred.ravel())
TN = cm[0][0]
FN = cm[1][0]
TP = cm[1][1]
FP = cm[0][1]
print(TN,FN,TP,FP)