0
$\begingroup$

I have been trying to fix this for 2 months now with no luck. I am doing some medical research for my study. I have a dataset that has patients diagnosis based on medical reports (Features.csv) and each patient based on that medical report has a list of diseases (Results.csv)

Here is a sample of each file.

Features.csv

filename    code    frequency
1006    53438000    2
1006    54706004    10
1006    65801008    1
1006    66842004    10
1006    70901006    11
1006    71388002    1
1006    71651007    1
1006    71960002    2
1006    73761001    2
1006    74016001    1
1006    77477000    1
1007    105011006   1
1007    34896006    1
1007    363680008   2
1007    399208008   2
1007    52765003    1
1007    57485005    1
1007    71388002    3
1007    73632009    1
1007    767002  1
1007    86273004    2
1008    34227000    1
1008    363679005   1
1008    42525009    1
1008    67166004    1
1008    71388002    1
1008    90205004    1
1009    104866001   1
1009    113011001   1
1009    113063008   1
1009    118635009   2
1009    122462000   1
1009    16310003    6
1009    165581004   1
1009    168537006   2
1009    169070004   1

This is Results.csv

filename    result  order
1006        5990    2
1006        7802    3
1006        2762    4
1006        2738    5
1006        4589    6
1006        V4575   7
1006        27651   8
1006        56400   9
1006        4019    10
1006        V103    11
1006        2449    12
1006        2724    13
1006        56210   14
1006        2859    15
1006        5779    16
1006        5566    1
1007        1892    1
1007        1970    2
1007        496 3
1007        4280    4
1007        51881   5
1007        2859    6
1007        4019    7
1007        V1011   8
1008        4321    1
1008        41400   2
1008        4019    3
1008        2724    4
1008        71590   5
1008        V4581   6
1009        0389    1
1009        5789    2
1009        5761    3
1009        5845    4
1009        51881   5
1009        1552    6
1009        5990    7
1009        4280    8
1009        5762    9
1009        57511   10
1009        25000   11
1009        V5867   12
1009        99592   13
1009        0413    14

I did this in my Python code

1- Load dataset.

2- Pivot features to have codes as columns and frequency is the value

filename   53438000   54706004 ... 90205004
1006       2          10           0
1007       0           0           0
1008       0           0           0
1009       0           0           0

3- Pivot Results and put values in array

1006 [5990, 7802, ...]
1007 [1892, 1970, ...]

4- One hot encode results

filename  1892   1970    .... 5990   7802 ...
1006       0     0            1       1
1007       1     1            0       0

5- Split dataset into Training and Test (80/20)

6- Use LogisticRegression

7- Check Accuracy

The accuracy I get is very very bad

and when I try to tweak code I get all 1s prediction!!

What am I doing wrong here?

How can I improve accuracy?

Here are the complete files.
Features
Results

and here is my code.

import pyodbc
import pandas as pd
import numpy as np
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier
from sklearn.metrics import f1_score
from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix


RecNo = "0"

pd.set_option('display.max_colwidth', 300)


conn = pyodbc.connect('Driver={SQL Server};'
                      'Server=DELLG3;'
                      'Database=NLP2;'
                      'Trusted_Connection=yes;')

df_features = pd.read_sql("EXEC GetFeatures " + RecNo , conn)


df_features.shape

df_results = pd.read_sql("EXEC GetResults " + RecNo  , conn)




df_features = df_features.pivot(index='filename', columns='code', values='frequency')
df_features[np.isnan(df_features)] = 0

df_results = df_results.groupby('filename')[["result"]].agg(list).reset_index()


multilabel_binarizer = MultiLabelBinarizer()
multilabel_binarizer.fit(df_results['result'])
ResultsArray = multilabel_binarizer.transform(df_results['result'])

ResultsArray = ResultsArray[:, :-1]

xtrain, xval, ytrain, yval = train_test_split(df_features, ResultsArray, test_size=0.2, random_state=9)

lr = LogisticRegression()
clf = OneVsRestClassifier(lr)


# fit model on train data
clf.fit(xtrain, ytrain)

# make predictions for validation set
y_pred = clf.predict(xval)

# evaluate performance
print(f1_score(yval, y_pred, average="micro"))

#I obtain the accuracy of this fold
ac=accuracy_score(y_pred,yval)

#I obtain the confusion matrix
cm=confusion_matrix(yval.ravel(), y_pred.ravel())

TN = cm[0][0]
FN = cm[1][0]
TP = cm[1][1]
FP = cm[0][1]
print(TN,FN,TP,FP)
Improve this question
$\endgroup$
3
  • $\begingroup$ Sorry, but what are you trying to predict? 'results'? What are your features? How did you merge Features.csv and Results.csv? Sorry it was not clear and I did not look at the code and your source. $\endgroup$
    – TwinPenguins
    Apr 22, 2020 at 7:06
  • $\begingroup$ Yes, Results is what I am trying to predict. my features are the codes in features.csv ... filename is the key between both files features and results. I have put all files in SQL Server database and call them from df_features = pd.read_sql("EXEC GetFeatures " + RecNo , conn) $\endgroup$
    – asmgx
    Apr 22, 2020 at 7:20
  • $\begingroup$ OK, understood, thanks. One more Q, have you tried other algos, like greedy ones like Gradient Boosting Trees (are prone to over-fitting, but it is OK to start), also your OHE could be problematic method too. Have you heard Catboost library? I would recommend training using Catboost that handles automatically the encoding part. I also suspect that is hard to build a good mapping between your feature and target, and you need higher level order functions with non-linearity elements (educated guessing here) $\endgroup$
    – TwinPenguins
    Apr 22, 2020 at 11:05

1 Answer 1

Reset to default
0
$\begingroup$

Looks filename is your dependent variable, and some features are multi label outputs, as in 4- One hot encode results.

If so, are you going to train model with filename or file contents. Can you please check on it.

Correct me if my understanding is wrong.

Specify about your X and y.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.