I am trying to classify texts into categories (one text can have multiple categories), to do this I used one-hot encoded labels (with sklearn.preprocessing.MultiLbaleBinarizer()). My texts are also one-hot encoded but with keras.preprocessing.text.one_hot.

This is my code :

csv_data_name = 'learning_base'
df = pd.read_csv('../../datasets/csv/' + csv_data_name + '.csv ')
df['label_list'] = np.array(df.labels.str.split('|'))
df = df.drop(columns=['labels'])

# Remove incomplete lines (lines without labels)
df = df.dropna()

encoder = MultiLabelBinarizer()
labels_binary = encoder.fit_transform(df.label_list)
output_dim = labels_binary.shape[1]

vocab_size = 200000
encoded_docs = [one_hot(d, vocab_size) for d in df.text]
max_length = 200
padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')

# Data division
training_set_data, testing_set_data_whole, training_set_target, testing_set_target_whole = train_test_split(

model = Sequential()
model.add(Embedding(vocab_size, 200, input_length=max_length))
model.add(Dense(64, activation='relu'))
model.add(Dense(output_dim, activation='sigmoid'))

model.compile(optimizer=TFOptimizer(tf.train.AdamOptimizer()), loss='mean_squared_error',

model.fit(training_set_data, training_set_target, epochs=50, verbose=True, batch_size=200,
          callbacks=[EarlyStopping(monitor='acc', patience=3, verbose=True), tensorboard])

Everything works fine except when I try another optimizer the learning seems incorrect, only Adam seems to work as intended.

This is my results :

|Optimizer|Time by epoch|Strict classification succes|Precision|Recall|epoch|
|-------- | ----------- |--------------------------- | ------- |------|---|
|GradientDescent                          |00:21|0%|12.5%|0%|3 (ES)|
|GradientDescent (`learning_rate = 0.01`) |00:21|0%|3.9%|16.1%|7 (ES)|
|Adadelta                                 |00:24|0%|3.1%|25.7%|3 (ES)|
|Adagrad                                  |00:26|0%|/|/|13 (ES)|
|Ftrl                                     |00:27|0%|/|/|8 (ES)|
|Momentum (`momentum = 0.01`)             |00:22|0%|/|/|18 (ES)|
|Adam (`learning_rate = 0.01` : default)  |02:37|53%|80%|60.54%|13 (ES)|

And here is the tensorboard for accuracy : tensorboard for accuracy plot


There could be a couple of possible reasons:

  1. One reason could be the Adam optimizer is a combination of several other optimization techniques (e.g., momentum and running average of gradient squares). The combination of those techniques works well on multi-label text classification.
  2. Another reason could be that multi-label text classification is a sparse problem. The Adam optimizer in TensorFlow has a sparse implementation.

The reason may be because the fixed learning rate you chose does not fit well with the data. If you try the Backtracking version, where learning rates are adapted at every step, and where convergence can be proven rigidly for many functions, things can be better. You can look at the paper in my answer in this link:

Does gradient descent always converge to an optimum?

Also, did you try NAG? In the paper I mentioned, we see from experiments that the Backtracking NAG works quite well and stable.


It probably doesn't. What you are comparing is not exactly fair. If you want to do it right, you should try to find the best hyper-parameters for all optimizers and compare their respective performance. What this tells you now is basically "which optimizer works best with the standard hyper-parameters and some arbitrary ones i chose".


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.