0
$\begingroup$

How can I get the size or the len of the dataset after applying a filter. Using tf.data.experimental.cardinality give -2, and this is not what I am looking for!! I want to know how many filtered samples exist in my dataset in order to be able to split it to training and validation datasets using take() and skip().

Example:

    dataset = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4, 5])
    dataset = dataset.filter(lambda x: x < 4)
    size = tf.data.experimental.cardinality(dataset).numpy()
    #size here is equal to -2 but I want to get the real size which is 3

My dataset contains images and their labels, this is just an illustrative example

$\endgroup$
2
  • $\begingroup$ If it is just a one time operation and the dataset size is not huge, you can use len(list(dataset)). $\endgroup$ – bkshi Jun 13 '20 at 17:03
  • $\begingroup$ Actually the dataset is huge, and I tried it but it took a long time (couldn't get the results because I had to stop the script) $\endgroup$ – Lukas Jun 13 '20 at 17:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.