0
$\begingroup$

I am new to deep learning. I have imbalanced class data. I used one hot encoding and scaling to preprocess my data. I have used adamoptimizer as optimizer function and sparse categorical crossentropy as my lass function. The model always gives high accuracy on one class with very low accuracy on other classes. Here is my code:

`

#separating test data according to classes
data_test = data_final[data_final.YEAR.isin(2018)]
data_test_0 = data_test[data_test['DELAY_CLASS']==0]
test_labels_0 = data_test_0.pop('DELAY_CLASS')
data_test_1 = data_test[data_test['DELAY_CLASS']==1]
test_labels_1 = data_test_1.pop('DELAY_CLASS')
data_test_2 = data_test[data_test['DELAY_CLASS']==2]
test_labels_2 = data_test_2.pop('DELAY_CLASS')
data_test_3 = data_test[data_test['DELAY_CLASS']==3]
test_labels_3 = data_test_3.pop('DELAY_CLASS')


#Extracting continuous columns from training data 
data_train = data_train[['MONTH','DAY_OF_MONTH','DAY_OF_WEEK',
'Dep_Hour','Arr_Hour','CRS_ELAPSED_TIME','DISTANCE','traffic','O_SurfaceTemperatureFahrenheit','O_CloudCoveragePercent','O_WindSpeedMph','O_PrecipitationPreviousHourInches','O_SnowfallInches','D_SurfaceTemperatureFahrenheit',
'D_CloudCoveragePercent','D_WindSpeedMph','D_PrecipitationPreviousHourInches','D_SnowfallInches','Bird_Strike']]

#Extracting continuous columns from testing data
data_test = data_test[['MONTH','DAY_OF_MONTH','DAY_OF_WEEK',
'Dep_Hour','Arr_Hour','CRS_ELAPSED_TIME','DISTANCE','traffic','O_SurfaceTemperatureFahrenheit','O_CloudCoveragePercent','O_WindSpeedMph','O_PrecipitationPreviousHourInches','O_SnowfallInches','D_SurfaceTemperatureFahrenheit',
'D_CloudCoveragePercent','D_WindSpeedMph','D_PrecipitationPreviousHourInches','D_SnowfallInches','Bird_Strike']]



print("reached here")

#SMOTE
sm = SMOTE(random_state=2)
ad = ADASYN(random_state=2)
data_train, train_labels = sm.fit_sample(data_train, train_labels)

data_train = pd.DataFrame(data_train)
data_train = data_train.rename(columns = {0:'MONTH',1:'DAY_OF_MONTH',2:'DAY_OF_WEEK',3:'Dep_Hour',
4:'Arr_Hour', 5:'CRS_ELAPSED_TIME', 6:'DISTANCE',
7:'traffic',8:'O_SurfaceTemperatureFahrenheit',9:'O_CloudCoveragePercent',
10:'O_WindSpeedMph',11:'O_PrecipitationPreviousHourInches',12:'O_SnowfallInches',
13:'D_SurfaceTemperatureFahrenheit',14:'D_CloudCoveragePercent',15:'D_WindSpeedMph',
16:'D_PrecipitationPreviousHourInches',17:'D_SnowfallInches',18:'Bird_Strike'})

#taking only continuous columns
cols = ['MONTH','DAY_OF_MONTH','DAY_OF_WEEK',
'Dep_Hour','Arr_Hour','CRS_ELAPSED_TIME','DISTANCE','traffic','O_SurfaceTemperatureFahrenheit','O_CloudCoveragePercent','O_WindSpeedMph','O_PrecipitationPreviousHourInches','O_SnowfallInches','D_SurfaceTemperatureFahrenheit','D_CloudCoveragePercent','D_WindSpeedMph','D_PrecipitationPreviousHourInches','D_SnowfallInches','Bird_Strike']

#scaling
train_mean = data_train[cols].mean(axis=0)
train_std = data_train[cols].std(axis=0)
data_train[cols] = (data_train[cols] - train_mean) / train_std
data_test[cols] = (data_test[cols] - train_mean) / train_std
rain_labels = pd.Series(train_labels)

#taking continuous columns from test separated data
data_test_0 = data_test_0[['MONTH','DAY_OF_MONTH','DAY_OF_WEEK',
'Dep_Hour','Arr_Hour','CRS_ELAPSED_TIME','DISTANCE','traffic','O_SurfaceTemperatureFahrenheit','O_CloudCoveragePercent','O_WindSpeedMph','O_PrecipitationPreviousHourInches','O_SnowfallInches','D_SurfaceTemperatureFahrenheit','D_CloudCoveragePercent','D_WindSpeedMph','D_PrecipitationPreviousHourInches','D_SnowfallInches','Bird_Strike']]

data_test_1 = data_test_1[['MONTH','DAY_OF_MONTH','DAY_OF_WEEK','Dep_Hour','Arr_Hour','CRS_ELAPSED_TIME','DISTANCE','traffic','O_SurfaceTemperatureFahrenheit','O_CloudCoveragePercent','O_WindSpeedMph','O_PrecipitationPreviousHourInches','O_SnowfallInches','D_SurfaceTemperatureFahrenheit','D_CloudCoveragePercent','D_WindSpeedMph','D_PrecipitationPreviousHourInches','D_SnowfallInches','Bird_Strike']]

data_test_2 = data_test_2[['MONTH','DAY_OF_MONTH','DAY_OF_WEEK',
'Dep_Hour','Arr_Hour','CRS_ELAPSED_TIME','DISTANCE','traffic','O_SurfaceTemperatureFahrenheit','O_CloudCoveragePercent','O_WindSpeedMph','O_PrecipitationPreviousHourInches','O_SnowfallInches','D_SurfaceTemperatureFahrenheit',
'D_CloudCoveragePercent','D_WindSpeedMph','D_PrecipitationPreviousHourInches','D_SnowfallInches','Bird_Strike']]

data_test_3 = data_test_3[['MONTH','DAY_OF_MONTH','DAY_OF_WEEK',
'Dep_Hour','Arr_Hour','CRS_ELAPSED_TIME','DISTANCE','traffic','O_SurfaceTemperatureFahrenheit','O_CloudCoveragePercent','O_WindSpeedMph','O_PrecipitationPreviousHourInches','O_SnowfallInches','D_SurfaceTemperatureFahrenheit',
'D_CloudCoveragePercent','D_WindSpeedMph','D_PrecipitationPreviousHourInches','D_SnowfallInches','Bird_Strike']]

#my model
def build_model():
    model = keras.Sequential([
        layers.Dense(100, activation = 'sigmoid', input_shape=[len(data_train.keys())]),
        #layers.Dropout(0.5),
        layers.Dense(50, activation = 'softplus'),
        #layers.Dropout(0.3),
        layers.Dense(25, activation = 'sigmoid'),
        #layers.Dropout(0.2),
        layers.Dense(4, activation = 'softmax')
    ])

    model.compile(loss='sparse_categorical_crossentropy',#with binary crossentropy use sigmoid and 1 output neuron
                  optimizer= tf.train.AdamOptimizer(0.001),
                  metrics=['accuracy'])
    return model

model = build_model()
model.fit(data_train, train_labels, epochs=5, batch_size=128)



test_loss, test_acc = model.evaluate(data_test_0, test_labels_0)
print(test_acc)
test_loss, test_acc = model.evaluate(data_test_1, test_labels_1)
print(test_acc)
test_loss, test_acc = model.evaluate(data_test_2, test_labels_2)
print(test_acc)
test_loss, test_acc = model.evaluate(data_test_3, test_labels_3)
print(test_acc)

`

The training data is flights data of 2016 and 2017 and testing data is of 2018. I have separated classes from testing data to see the class wise accuracy of testing data.

The output is:

Epoch 1/5
1990363/1990363 [==============================] - 17s 8us/step - loss: 1.3231 - acc: 0.3466
Epoch 2/5
1990363/1990363 [==============================] - 17s 8us/step - loss: 1.2799 - acc: 0.3821
Epoch 3/5
1990363/1990363 [==============================] - 17s 8us/step - loss: 1.2634 - acc: 0.3939
Epoch 4/5
1990363/1990363 [==============================] - 17s 8us/step - loss: 1.2519 - acc: 0.4013
Epoch 5/5
1990363/1990363 [==============================] - 16s 8us/step - loss: 1.2445 - acc: 0.4068

Class 0: 
44929/44929 [==============================] - 1s 12us/step
0.027710387500278218
Class 1:
10668/10668 [==============================] - 0s 11us/step
0.015935508061492312
Class 2:
33204/33204 [==============================] - 0s 9us/step
0.8956149861318866
Class 3:
274983/274983 [==============================] - 2s 9us/step
0.035293090845941046

The output remains somewhat same if I use adasyn instead of SMOTE or change layers and activation functions. Please help me out. Thanks in advance.

$\endgroup$
  • $\begingroup$ Hard to say what's happening... I would suggest using tensorboard so that you can visualize what's happening within your model. Try running it again with tensorboard and then post your histogram visualizations $\endgroup$ – Leo Brueggeman Apr 18 '19 at 22:39
  • $\begingroup$ @LeoBrueggeman , Ser, My model is built on Keras. Is it possible to see it on tensorboard? $\endgroup$ – Bhupesh_decoder Apr 19 '19 at 5:09
  • $\begingroup$ Yeah, it's possible. See this comment for details on how to do it. stackoverflow.com/a/42112935/10112380. I would change histogram_freq = 1. $\endgroup$ – Leo Brueggeman Apr 19 '19 at 14:35
  • $\begingroup$ Do you have baseline results using decision trees or regression? It would allow you to see if the problem is with your data or your model. $\endgroup$ – Yvonne Sep 5 '19 at 9:37
  • $\begingroup$ is the typo in rain_labels in your actual code? -> rain_labels = pd.Series(train_labels) $\endgroup$ – redhqs Sep 5 '19 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.