# Multiclass classification dataset with many features producing bad accuracy of predictions

I have been trying to fix this for 2 months now with no luck. I am doing some medical research for my study. I have a dataset that has patients diagnosis based on medical reports (Features.csv) and each patient based on that medical report has a list of diseases (Results.csv)

Here is a sample of each file.

Features.csv

filename    code    frequency
1006    53438000    2
1006    54706004    10
1006    65801008    1
1006    66842004    10
1006    70901006    11
1006    71388002    1
1006    71651007    1
1006    71960002    2
1006    73761001    2
1006    74016001    1
1006    77477000    1
1007    105011006   1
1007    34896006    1
1007    363680008   2
1007    399208008   2
1007    52765003    1
1007    57485005    1
1007    71388002    3
1007    73632009    1
1007    767002  1
1007    86273004    2
1008    34227000    1
1008    363679005   1
1008    42525009    1
1008    67166004    1
1008    71388002    1
1008    90205004    1
1009    104866001   1
1009    113011001   1
1009    113063008   1
1009    118635009   2
1009    122462000   1
1009    16310003    6
1009    165581004   1
1009    168537006   2
1009    169070004   1


This is Results.csv

filename    result  order
1006        5990    2
1006        7802    3
1006        2762    4
1006        2738    5
1006        4589    6
1006        V4575   7
1006        27651   8
1006        56400   9
1006        4019    10
1006        V103    11
1006        2449    12
1006        2724    13
1006        56210   14
1006        2859    15
1006        5779    16
1006        5566    1
1007        1892    1
1007        1970    2
1007        496 3
1007        4280    4
1007        51881   5
1007        2859    6
1007        4019    7
1007        V1011   8
1008        4321    1
1008        41400   2
1008        4019    3
1008        2724    4
1008        71590   5
1008        V4581   6
1009        0389    1
1009        5789    2
1009        5761    3
1009        5845    4
1009        51881   5
1009        1552    6
1009        5990    7
1009        4280    8
1009        5762    9
1009        57511   10
1009        25000   11
1009        V5867   12
1009        99592   13
1009        0413    14


I did this in my Python code

2- Pivot features to have codes as columns and frequency is the value

filename   53438000   54706004 ... 90205004
1006       2          10           0
1007       0           0           0
1008       0           0           0
1009       0           0           0


3- Pivot Results and put values in array

1006 [5990, 7802, ...]
1007 [1892, 1970, ...]


4- One hot encode results

filename  1892   1970    .... 5990   7802 ...
1006       0     0            1       1
1007       1     1            0       0


5- Split dataset into Training and Test (80/20)

6- Use LogisticRegression

7- Check Accuracy

The accuracy I get is very very bad

and when I try to tweak code I get all 1s prediction!!

What am I doing wrong here?

How can I improve accuracy?

Here are the complete files.
Features
Results

and here is my code.

import pyodbc
import pandas as pd
import numpy as np
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier
from sklearn.metrics import f1_score
from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix

RecNo = "0"

pd.set_option('display.max_colwidth', 300)

conn = pyodbc.connect('Driver={SQL Server};'
'Server=DELLG3;'
'Database=NLP2;'
'Trusted_Connection=yes;')

df_features = pd.read_sql("EXEC GetFeatures " + RecNo , conn)

df_features.shape

df_results = pd.read_sql("EXEC GetResults " + RecNo  , conn)

df_features = df_features.pivot(index='filename', columns='code', values='frequency')
df_features[np.isnan(df_features)] = 0

df_results = df_results.groupby('filename')[["result"]].agg(list).reset_index()

multilabel_binarizer = MultiLabelBinarizer()
multilabel_binarizer.fit(df_results['result'])
ResultsArray = multilabel_binarizer.transform(df_results['result'])

ResultsArray = ResultsArray[:, :-1]

xtrain, xval, ytrain, yval = train_test_split(df_features, ResultsArray, test_size=0.2, random_state=9)

lr = LogisticRegression()
clf = OneVsRestClassifier(lr)

# fit model on train data
clf.fit(xtrain, ytrain)

# make predictions for validation set
y_pred = clf.predict(xval)

# evaluate performance
print(f1_score(yval, y_pred, average="micro"))

#I obtain the accuracy of this fold
ac=accuracy_score(y_pred,yval)

#I obtain the confusion matrix
cm=confusion_matrix(yval.ravel(), y_pred.ravel())

TN = cm[0][0]
FN = cm[1][0]
TP = cm[1][1]
FP = cm[0][1]
print(TN,FN,TP,FP)

• Sorry, but what are you trying to predict? 'results'? What are your features? How did you merge Features.csv and Results.csv? Sorry it was not clear and I did not look at the code and your source. – TwinPenguins Apr 22 '20 at 7:06
• Yes, Results is what I am trying to predict. my features are the codes in features.csv ... filename is the key between both files features and results. I have put all files in SQL Server database and call them from df_features = pd.read_sql("EXEC GetFeatures " + RecNo , conn) – asmgx Apr 22 '20 at 7:20
• OK, understood, thanks. One more Q, have you tried other algos, like greedy ones like Gradient Boosting Trees (are prone to over-fitting, but it is OK to start), also your OHE could be problematic method too. Have you heard Catboost library? I would recommend training using Catboost that handles automatically the encoding part. I also suspect that is hard to build a good mapping between your feature and target, and you need higher level order functions with non-linearity elements (educated guessing here) – TwinPenguins Apr 22 '20 at 11:05

Looks filename is your dependent variable, and some features are multi label outputs, as in 4- One hot encode results.
If so, are you going to train model with filename or file contents. Can you please check on it.