1
$\begingroup$

Ive been thinking about combining some processes between keras and Sci-kit Learn and am looking to the this group to either validate my process or tell Im crazy. Im creating a simple Regression problem using 17 inputs like this:

Creating test/train here:

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.25, random_state=7)

Building the network here:

model = Sequential()
model.add(Dense(34, input_dim=17, kernel_initializer='normal', activation='relu'))
model.add(Dense(17, kernel_initializer='normal', activation='relu'))
model.add(Dense(8, kernel_initializer='normal'))
model.add(Dense(1, kernel_initializer='normal'))

compiling the model with:

model.compile(loss='mean_squared_error', optimizer='adam')

fitting the model here:

model.fit(X_train,y_train,validation_data=(X_test,y_test), epochs=100, batch_size=10)

Now that Ive fit the model is the any reason I can use some of the SciKit functions and do the following?

Make predictions

y_pred = model.predict(X_test)

Assess the model results:

mse = mean_squared_error(y_test, y_pred)
rmse = sqrt(mse)
r2score = r2_score(y_test,y_pred)

Am I way off-base here?

$\endgroup$
0
$\begingroup$

This seems about right.

You can use SciKit learn quite easily, as the predictions and test results you have should all be in NumPy arrays anyway. Take a look at the regression metrics. The metrics you have named are shown in the documentation to take lists, but here is an example showing they work just as well with NumPy arrays:

In [1]: from sklearn.metrics import r2_score

In [2]: y_true = [3, -0.5, 2, 7]

In [3]: y_pred = [2.5, 0.0, 2, 8]

In [4]: r2_score(y_true, y_pred)
Out[4]: 0.94860813704496794

In [6]: import numpy as np

In [13]: y_true = np.array([3, -0.5, 2, 7])

In [14]: y_pred = np.array([2.5, 0.0, 2, 8])

In [15]: r2_score(y_true, y_pred)
Out[15]: 0.94860813704496794            # identical result

One last comment:

17 examples doesn't sound like a lot. Look at model.summary() after compilation to see how many parameters your model has (spoiler - it's 1360). I would expect that you model (with your number of epochs and batch size etc.) will overfit, just memorising the dataset, and probably score 100%.

While this is a good sanity check to make sure your model can indeed learn, it might be a good idea to split a larger dataset (if available) into train, validation and test datasets. The simply use train and val as you have above, but in the prediction line, us the test set, which the model has never seen before. Unless your data is extremely homogenous, I wouldn't expect a accuracy metric near 100%.

$\endgroup$
2
  • $\begingroup$ Thanks for the comments they're very helpful. As far as the 17 goes, those are my input features and not the number of observations. I have over 10k observations in my training set. $\endgroup$ – DataGuy Jun 11 '18 at 15:09
  • $\begingroup$ @DataGuy - glad it helped and sorry for misunderstanding you! $\endgroup$ – n1k31t4 Jun 11 '18 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.