You can see in the fit()
method on your Model
instance, that the input is first sent through a method called _standardize_user_data
at line 643.
Source of error
Your error message comes from the checks that happen across lines 650 to 671 in that file:
if not self.built:
# We need to use `x` to set the model inputs.
# We type-check that `x` and `y` are either single arrays
# or lists of arrays.
if isinstance(x, (list, tuple)):
if not all(isinstance(v, np.ndarray) or
K.is_tensor(v) for v in x):
raise ValueError('Please provide as model inputs '
'either a single '
'array or a list of arrays. '
'You passed: x=' + str(x))
all_inputs += list(x)
elif isinstance(x, dict):
raise ValueError('Please do not pass a dictionary '
'as model inputs.')
else:
if not isinstance(x, np.ndarray) and not K.is_tensor(x):
raise ValueError('Please provide as model inputs '
'either a single '
'array or a list of arrays. '
'You passed: x=' + str(x))
all_inputs.append(x)
They use isinstance()
to check the type, and your HDF5 type is not covered anywhere.
Possible hack
I linked you GitHub issue as a comment on your post. Hpowever...
You could alter the code above in your local version of Keras to cover your case, essentially converting the received input into a NumPy array, which would then pass then checks and be used.
I would probably just enter a second elif
to the conditions above, like this:
elif isinstance(x, dict):
raise ValueError('Please do not pass a dictionary '
'as model inputs.') # original code
#### add this snippet #############
elif isinstance(x, h5py._hl.dataset.Dataset):
x = np.array(x) # you might need to find a more elegant way of converting the HDF5 block to a numpy array
###################################
else:
if not isinstance(x, np.ndarray) and not K.is_tensor(x): # original code
You can confirm that the correct type of data for you is that h5py._hl.dataset.Dataset
by checking the output of type(keras.utils.io_utils.HDF5Matrix('dataset.h5', 'x_train'))
.
This should get things working, although it might cost you some of the other benefits of the HDF5 loading system, such as specifying start
and end
indices.
Testing
Just an example to show how the above transformation should really end up with your data being fed to the model as a numpy array:
import numpy as np
import keras
a = np.arange(0, 75).reshape((5, 5, 3)) # like a 5x5 RGB image
f = h5py.File('tester.h5', 'w')
f.create_dataset(name='a', data=a)
# output: <HDF5 dataset "a": shape (5, 5), type "<i8">
f.close()
# Later on ...
data = h5py.File('trial.h5', 'r')
data = np.array(data['a'])
np.array_equal(data, a)
# True
Given the documentation on the the HDF5 utility:
keras.utils.HDF5Matrix(datapath, dataset, start=0, end=None, normalizer=None)
Representation of HDF5 dataset to be used instead of a Numpy array.
it does feel like there is a bug, or at least a discrepancy between the documentation and the code.
EDIT
You can fit via custom generator that would load blocks from your HDF5Matrix.
def generate_arrays_from_file(path):
while True:
with h5py.File(path) as f:
for batch in f:
# read the data and reshape as necessary
trainX, trainY, testX, testY = split_batch(batch)
yield (trainX, trainY)
model.fit_generator(generate_arrays_from_file('dataset.h5'),
steps_per_epoch=100, epochs=10)
You will have to obviously write a generator function yourself that matches you exact h5 format. Perhaps have a look at the fancy indexing options of h5 files.
conda list
in terminal, using the same environment as your model. $\endgroup$