I want to try out Keras (Theano backend) for regressions after already using sklearn.

For this I uses this nice tutorial http://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/ and tried to replace the training data there with my own.

import numpy
import pandas
import pickle
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

from sklearn.model_selection import train_test_split

from sklearn.preprocessing import StandardScaler

[X, Y] = pickle.load(open("training_data_1_week_imp_lt_15.pkl", "rb"))

X_train, X_test, y_train, y_test = train_test_split( X, Y, test_size=0.5, random_state=42)
scaler = StandardScaler()
scaler.fit(X_train)  # Don't cheat - fit only on training data
X_train = scaler.transform(X_train)

X_test = scaler.transform(X_test)

print (X_train.shape)

# define base mode
def baseline_model():
    # create model
    model = Sequential()
    model.add(Dense(8, input_dim=8, init='normal', activation='relu'))
    model.add(Dense(1, init='normal'))
    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

# fix random seed for reproducibility
seed = 7
# evaluate model with standardized dataset
estimator = KerasRegressor(build_fn=baseline_model, nb_epoch=100, batch_size=5, verbose=0)    


However, i get the following error:

Exception: Error when checking model target: the list of Numpy arrays that you are 
passing to your model is not the size the model expected. Expected to see 1 
arrays but instead got the following list of 6252 arrays: ...

The format of X is the usual sklearn format: print (X_train.shape) = (6252, 8)

How do I format my input X correctly.

What I tried was transposing but this did not work.

I also already searched the web but could not find a solution/explanation.


EDIT: here is a small sample file https://ufile.io/8a428

[X, Y] = pickle.load(open("test.pkl", "rb"))
  • $\begingroup$ It seems your X_train is a python list which contains numpy ndarrays, rather than a single numpy ndarray. Could you please upload your training_data_1_week_imp_lt_15.pkl? $\endgroup$
    – Icyblade
    Feb 21 '17 at 10:24
  • $\begingroup$ Yes it is: X is essentially a list of lists. $\endgroup$
    – El Burro
    Feb 21 '17 at 10:29
  • $\begingroup$ However, I transform it to an ndarray estimator.fit(numpy.array(X_train),y_train) or at least thats what i thought i did $\endgroup$
    – El Burro
    Feb 21 '17 at 10:52
  • $\begingroup$ In my environment, your original code works with the sample data you uploaded. Maybe you can try to reinstall numpy? $\endgroup$
    – Icyblade
    Feb 21 '17 at 13:30
  • $\begingroup$ Despite the fix- it is interesting that it worked for you without. $\endgroup$
    – El Burro
    Feb 21 '17 at 14:58

I solved this (still banging my head against the wall):


this works. I am not sure why. The error message is very misleading IMHO.

  • 1
    $\begingroup$ The ... part(the part you omitted) of your error is your y_train data right? It may explain why np.array(y_train) works. $\endgroup$
    – Icyblade
    Feb 21 '17 at 14:04
  • $\begingroup$ Yes. Thank you anyway- your questions made me solve this after really being stuck :). $\endgroup$
    – El Burro
    Feb 21 '17 at 14:57
  • $\begingroup$ The interesting part is that X_train and y_train are generated from train_test_split, which will return numpy ndarrays for sure. I can't figure out why your X_train or y_train is a list. $\endgroup$
    – Icyblade
    Feb 21 '17 at 15:02
  • $\begingroup$ neither can I - especially if one considers the fact that the exact same setting works for pure sklearn problems. $\endgroup$
    – El Burro
    Feb 22 '17 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.