Making predictions from keras with SciKit

Question

Ive been thinking about combining some processes between keras and Sci-kit Learn and am looking to the this group to either validate my process or tell Im crazy. Im creating a simple Regression problem using 17 inputs like this:

Creating test/train here:

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.25, random_state=7)

Building the network here:

model = Sequential()
model.add(Dense(34, input_dim=17, kernel_initializer='normal', activation='relu'))
model.add(Dense(17, kernel_initializer='normal', activation='relu'))
model.add(Dense(8, kernel_initializer='normal'))
model.add(Dense(1, kernel_initializer='normal'))

compiling the model with:

model.compile(loss='mean_squared_error', optimizer='adam')

fitting the model here:

model.fit(X_train,y_train,validation_data=(X_test,y_test), epochs=100, batch_size=10)

Now that Ive fit the model is the any reason I can use some of the SciKit functions and do the following?

Make predictions

y_pred = model.predict(X_test)

Assess the model results:

mse = mean_squared_error(y_test, y_pred)
rmse = sqrt(mse)
r2score = r2_score(y_test,y_pred)

Am I way off-base here?

Answer 1

This seems about right.

You can use SciKit learn quite easily, as the predictions and test results you have should all be in NumPy arrays anyway. Take a look at the regression metrics. The metrics you have named are shown in the documentation to take lists, but here is an example showing they work just as well with NumPy arrays:

In [1]: from sklearn.metrics import r2_score

In [2]: y_true = [3, -0.5, 2, 7]

In [3]: y_pred = [2.5, 0.0, 2, 8]

In [4]: r2_score(y_true, y_pred)
Out[4]: 0.94860813704496794

In [6]: import numpy as np

In [13]: y_true = np.array([3, -0.5, 2, 7])

In [14]: y_pred = np.array([2.5, 0.0, 2, 8])

In [15]: r2_score(y_true, y_pred)
Out[15]: 0.94860813704496794            # identical result

---

One last comment:

17 examples doesn't sound like a lot. Look at model.summary() after compilation to see how many parameters your model has (spoiler - it's 1360). I would expect that you model (with your number of epochs and batch size etc.) will overfit, just memorising the dataset, and probably score 100%.

While this is a good sanity check to make sure your model can indeed learn, it might be a good idea to split a larger dataset (if available) into train, validation and test datasets. The simply use train and val as you have above, but in the prediction line, us the test set, which the model has never seen before. Unless your data is extremely homogenous, I wouldn't expect a accuracy metric near 100%.