Return to Answer

I would use tensorflow 2.0 with tf.data

import tensorflow as tf

filenames = ["filename1", "filename2", ...]

dataset = tf.data.Dataset.list_files(filenames, seed=42, shuffle=True)

# this reads 5 text files at a time, skips the first row of each file
dataset.interleave(lambda filename: tf.data.TextLineDataset(filename).skip(1), cycle_length=5, num_parallel_calls=tf.data.experimental.AUTOTUNE)

for line in dataset.take(5):
    print(line)

You can then pass in the dataset objects directly into tensorflow.keras for training.

There are some good examples here: https://cs230-stanford.github.io/tensorflow-input-data.html but please beware that this is NOT for tensorflow 2.0, and somethings may have changed

I would use tensorflow 2.0 with tf.data

import tensorflow as tf

filenames = ["filename1", "filename2", ...]

dataset = tf.data.Dataset.list_files(filenames, seed=42, shuffle=True)

# this reads 5 text files at a time, skips the first row of each file
dataset.interleave(lambda filename: tf.data.TextLineDataset(filename).skip(1), cycle_length=5, num_parallel_calls=tf.data.experimental.AUTOTUNE)

for line in dataset.take(5):
    print(line)

You can then pass in the dataset objects directly into tensorflow.keras for training.

There are some good examples here but please beware that this is NOT for tensorflow 2.0, and somethings may have changed

I would you use tensorflow 2.0 and tf.data

import tensorflow as tf

filenames = ["filename1", "filename2", ...]

dataset = tf.data.Dataset.list_files(filenames, seed=42, shuffle=True)

# this reads 5 text files at a time, skips the first row of each file
dataset.interleave(lambda filename: tf.data.TextLineDataset(filename).skip(1), cycle_length=5, num_parallel_calls=tf.data.experimental.AUTOTUNE)

for line in dataset.take(5):
    print(line)

You can then pass in the dataset objects directly into tensorflow.keras for training.

There are some good examples here: https://cs230-stanford.github.io/tensorflow-input-data.html but please beware that this is NOT for tensorflow 2.0, and somethings may have changed

I would use tensorflow 2.0 with tf.data

import tensorflow as tf

filenames = ["filename1", "filename2", ...]

dataset = tf.data.Dataset.list_files(filenames, seed=42, shuffle=True)

# this reads 5 text files at a time, skips the first row of each file
dataset.interleave(lambda filename: tf.data.TextLineDataset(filename).skip(1), cycle_length=5, num_parallel_calls=tf.data.experimental.AUTOTUNE)

for line in dataset.take(5):
    print(line)

You can then pass in the dataset objects directly into tensorflow.keras for training.

There are some good examples here: https://cs230-stanford.github.io/tensorflow-input-data.html but please beware that this is NOT for tensorflow 2.0, and somethings may have changed

I would you use tensorflow 2.0 and tf.data

import tensorflow as tf

filenames = ["filename1", "filename2", ...]

dataset = tf.data.Dataset.list_files(filenames, seed=42, shuffle=True)

# this reads 5 text files at a time, skips the first row of each file
dataset.interleave(lambda filename: tf.data.TextLineDataset(filename).skip(1), cycle_length=5, num_parallel_calls=tf.data.experimental.AUTOTUNE)

for line in dataset.take(5):
    print(line)

You can then pass in the dataset objects directly into tensorflow.keras for training.

I would you use tensorflow 2.0 and tf.data

import tensorflow as tf

filenames = ["filename1", "filename2", ...]

dataset = tf.data.Dataset.list_files(filenames, seed=42, shuffle=True)

# this reads 5 text files at a time, skips the first row of each file
dataset.interleave(lambda filename: tf.data.TextLineDataset(filename).skip(1), cycle_length=5, num_parallel_calls=tf.data.experimental.AUTOTUNE)

for line in dataset.take(5):
    print(line)

You can then pass in the dataset objects directly into tensorflow.keras for training.

There are some good examples here: https://cs230-stanford.github.io/tensorflow-input-data.html but please beware that this is NOT for tensorflow 2.0, and somethings may have changed