2
$\begingroup$

Hello and thank you in advance for any answer. I am building a NN for a multi class classification problem. I have pretrained my data through a Word2vec and generated a 300 dimension vector with values. I have 6 classes .. My data consist of 15000 rows(x300 dimensions). My first question is :

What is the number of units? Is it something that we can extract from theory? Also i have managed to gain 0.85-0.90 training set accuracy but the validation set accuracy is always low: 0.22-0.25. I do not know what way to follow as I am newbie to these kind of stuff. My code is:

import numpy as np
import pandas as pd
import keras
from sklearn.preprocessing import Imputer
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers import Dense,Dropout
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
from keras import regularizers



dataset = pd.read_csv('word2vec.csv')


X = dataset.iloc[:, 0:300].values
y = dataset.iloc[:, 300].values

imputer = Imputer(missing_values = 'NaN', strategy = 'mean', axis = 0)
imputer = imputer.fit(X[:, 0:300])
X[:, 0:300] = imputer.transform(X[:, 0:300])

counter = 0
for iterations in range(1):

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

    classifier = Sequential()

    classifier.add(Dense(units=70, kernel_initializer='uniform',kernel_regularizer=regularizers.l2(0.0001),
                activity_regularizer=regularizers.l1(0.001), activation='relu', input_dim=300))

    classifier.add(Dropout(0.3))

    # classifier.add(Dense(units=70, kernel_initializer='uniform', activation='relu'))


    classifier.add(Dense(units=6, kernel_initializer='uniform', activation='softmax'))

    classifier.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

    history = classifier.fit(X_train, y_train,validation_split=0.1, batch_size=10, epochs=100,shuffle=True)

    y_pred = classifier.predict(X_test)
    # y_pred = (y_pred > 0.1666)

    classifier.summary()
    counter+=1
    print(classifier.layers)
    print("Running RNN with Dropout Layer")
    print("Number of layers used: "+str(len(classifier.layers)))
    # list all data in history
    print(history.history.keys())
    # summarize history for accuracy
    plt.plot(history.history['acc'])
    plt.plot(history.history['val_acc'])
    plt.title('model accuracy')
    plt.ylabel('accuracy')
    plt.xlabel('epoch')
    plt.legend(['train', 'test'], loc='upper left')
    plt.show()
    # summarize history for loss
    plt.plot(history.history['loss'])
    plt.plot(history.history['val_loss'])
    plt.title('model loss')
    plt.ylabel('loss')
    plt.xlabel('epoch')
    plt.legend(['train', 'test'], loc='upper left')
    plt.show()
$\endgroup$
0
$\begingroup$

Yep - you've overfitted !

Can you see what your accuracy is as a function of the number of training cycles? It would be interesting to see how that looks.

Is there any way you could increase your data? 15000x300 isn't a huge amount of depth for a neural network, considering the width of the data set.

You could look at the impact of increasing your regularisation parameters, or decreasing the depth of your network

$\endgroup$
  • $\begingroup$ UPDATE: Thank you for your time.. Can you please explain to me about the accuracy and the training cycles? Unfortunately I cannot increase the the data.. But i could decrease the dimensions.. Do you think it would be helpful? Does this overfitting occurs mainly because of the low amount of data? Is my code wrong?? Thank you $\endgroup$ – Marinos Zagkotsis Nov 12 '18 at 11:13
  • $\begingroup$ @marinos in a general sense overfitting increases with model complexity, so reducing dimensions would be a good start. $\endgroup$ – davmor Nov 13 '18 at 5:50
  • $\begingroup$ @davmor thank you for the answer and your time.. I will implement it that way... I have applied several other methods such as l1,l2 regularizations, batch normalization and dropout and also split the validation set like i have for test set.. I managed to overcome the overfit.. now ofcourse my training accuracy is lower . Is it maybe because of the word2vec pretraining? maybe it comes in addition with the 300 dimensions?\ thank you $\endgroup$ – Marinos Zagkotsis Nov 13 '18 at 7:57
  • $\begingroup$ @marinos again, generally, modeling is about choosing trade offs. This is ok, as you will find most clients (and good managers) are fine if you explain the error and why (here, a small training set relative to dimensions). Secondly, analysis is a process, not a single event. So use your model to run new data, determine error, and improve. Lastly, remember in most real world cases, the current state is no model at all, so something is usually better than nothing. $\endgroup$ – davmor Nov 16 '18 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.