Scikit-learn and TensorFlow with very different MLP models

Question

I'm using Multilayer Perceptron ANNs at the very beginning of my project (it's a binary classification problem). Because it's simpler, I started with Scikit-learn. I got a magic result, with my model classifying correctly 98% of the test set. Now, I want to move to TensorFlow so I can implement other details, use dropout, and maybe other ANN architectures. The problem is: I tried to replicate the same ANN from sklearn in tensorflow, but now my score is 50% (just predicting everything as 1).

In both code excerpts below, I've used the same pre-treatment for both, having X_train and y_train as training data and X_test and y_test as testing data. As it's a binary classification problem, y_* is a vector of 1s and 0s.

This is my code in sklearn, which prints "Score of the prediction: 0.98 after 373 iterations.":

# ...
clf = MLPClassifier(max_iter=1000, hidden_layer_sizes=(100, 100))
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)

print(f"Score of the prediction: {clf.score(X_test, y_test)} after {clf.n_iter_} iterations.")

clf.predict(X_test), in this case, is a numpy.ndarray(dtype=int32), having only 1s and 0s, just like my original vector y_test.

In TensorFlow (with Keras), this is my code:

# ...
model = models.Sequential()
model.add(layers.Dense(100, activation='relu', input_shape=(X_train.shape[1],)))
model.add(layers.Dense(100, activation='relu'))
model.add(layers.Dense(1, activation='softmax'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X_train, y_train, batch_size=min([200, X.shape[0]]), validation_split=0.1)
test_results = model.evaluate(X_test, y_test, verbose=1)

And it prints:

1/1 [==============================] - 2s 2s/step - loss: 1.5126 - accuracy: 0.4889 - val_loss: 1.8084 - val_accuracy: 0.6000

Here, a val_accuracy of 0.6 is not much better than predicting everything as 1 (or everything as 0). It happens to be the case. model.predict(X_test) is a numpy.ndarray(dtype=float32) equals to [0.99999994, 0.99999994, ..., 0.99999994, 1].

Looking at the documentation for sklearn.neural_network.MLPClassifier, I found that it's using batchsize=min([200, X.shape[0]]), activation='relu' and optmizer='adam', exactly like I (tried to) mimick in TF.

The question is: What am I missing?

Note: I'm sorry for not providing the datasets, but it's private. However, being the same thing for both libraries, the focus is on the difference.

Answer 1

I know that your question was asked almost a year ago, but still maybe someone will find it useful. There are two problems:

The first is you are using softmax activation, yet only have one output neuron. When using softmax you need as many output neurons as you expect classes! Use sigmoid instead.

Another major problem is the discrapancy between the learning epochs. In the MLPClassifier you give the max_iter=1000, yet in tensorflow the default number of epochs while calling model.fit is just 1. Set it to epochs=1000 and it should be already better.

I am struggling myself to reimplement the MLPClassifier in tensorflow. I also used the L2 Regularization and it turns out it isn't as straightforward as it would seem to be. The regularization is only used on hidden layers and the alpha parameter from scikit-learn is divided by 2 before being used in the loss function. Acccording to source code for scikit-learn MLPClassifier:

 n_samples = X.shape[0]

 # Add L2 regularization term to loss
 values = 0
 for s in self.coefs_:
     s = s.ravel()
     values += np.dot(s, s)
 loss += (0.5 * self.alpha) * values / n_samples

Therefore for alpha from scikitlearn, in tensorflow use kernel_regularizer = tf.keras.regularizers.l2(0.5 * alpha) on your hidden layers

I still haven't achieved the exact same results, so beware that my hints are not comperhensive