# Train on batches in Tensorflow

I'm currently trying to train a model on a large csv file (>70GB with more than 60 million rows). To do so I'm using tf.contrib.learn.read_batch_examples. I'm struggling in understanding how this function actually reads the data. If I'm using a batch size of e.g. 50.000, does it read the first 50.000 lines of the file? If I want to loop over the whole file (1 epoch) do I have to use num_rows/batch_size = 1.200 number of steps for the estimator.fit method?

Here is the input function im currently using:

def input_fn(file_names, batch_size):
# Read csv files and create examples dict

# Continuous features
feature_cols = {k: tf.string_to_number(examples_dict[k],
out_type=tf.float32) for k in CONTINUOUS_COLUMNS}

# Categorical features
feature_cols.update({
k: tf.SparseTensor(
indices=[[i, 0] for i in range(examples_dict[k].get_shape()[0])],
values=examples_dict[k],
shape=[int(examples_dict[k].get_shape()[0]), 1])
for k in CATEGORICAL_COLUMNS})

label = tf.string_to_number(examples_dict[LABEL_COLUMN], out_type=tf.int32)

return feature_cols, label

def parse_fn(record):
record_defaults = [tf.constant([''], dtype=tf.string)] * len(COLUMNS)

return tf.decode_csv(record, record_defaults)

file_names,
batch_size=batch_size,
queue_capacity=batch_size*2.5,
parse_fn=parse_fn,
#randomize_input=True,
)

# Important: convert examples to dict for ease of use in input_fn
# Map each header to its respective column (COLUMNS order
# matters!
examples_dict_op = {}

return examples_dict_op


Here is the code im using to train the model:

def train_and_eval():
"""Train and evaluate the model."""

m = build_estimator(model_dir)
m.fit(input_fn=lambda: input_fn(train_file_name, batch_size), steps=steps)


What would happen if I would call the fit function again with the same input_fn. Does it start at the beginning of the file again, or will it remember the line where it has stopped last time?

Generally speaking a batch uses n times a record or item. How you define an item depends on your problem. In tensorflow the batch is encoded in the first dimension of a tensor. In your case with the csv file it might be line by line (reader=tf.TextLineReader). It could learn by column but I don't think that this is happening in your code. If you want train with your whole dataset (=one epoch) you can do so by using numBatches=numItems/batchSize.