12
$\begingroup$

I'm currently trying to train a model on a large csv file (>70GB with more than 60 million rows). To do so I'm using tf.contrib.learn.read_batch_examples. I'm struggling in understanding how this function actually reads the data. If I'm using a batch size of e.g. 50.000, does it read the first 50.000 lines of the file? If I want to loop over the whole file (1 epoch) do I have to use num_rows/batch_size = 1.200 number of steps for the estimator.fit method?

Here is the input function im currently using:

def input_fn(file_names, batch_size):
    # Read csv files and create examples dict
    examples_dict = read_csv_examples(file_names, batch_size)

    # Continuous features
    feature_cols = {k: tf.string_to_number(examples_dict[k],
                                           out_type=tf.float32) for k in CONTINUOUS_COLUMNS}

    # Categorical features
    feature_cols.update({
                            k: tf.SparseTensor(
                                indices=[[i, 0] for i in range(examples_dict[k].get_shape()[0])],
                                values=examples_dict[k],
                                shape=[int(examples_dict[k].get_shape()[0]), 1])
                            for k in CATEGORICAL_COLUMNS})

    label = tf.string_to_number(examples_dict[LABEL_COLUMN], out_type=tf.int32)

    return feature_cols, label


def read_csv_examples(file_names, batch_size):
    def parse_fn(record):
        record_defaults = [tf.constant([''], dtype=tf.string)] * len(COLUMNS)

        return tf.decode_csv(record, record_defaults)

    examples_op = tf.contrib.learn.read_batch_examples(
        file_names,
        batch_size=batch_size,
        queue_capacity=batch_size*2.5,
        reader=tf.TextLineReader,
        parse_fn=parse_fn,
        #read_batch_size= batch_size,
        #randomize_input=True,
        num_threads=8
    )

    # Important: convert examples to dict for ease of use in `input_fn`
    # Map each header to its respective column (COLUMNS order
    # matters!
    examples_dict_op = {}
    for i, header in enumerate(COLUMNS):
        examples_dict_op[header] = examples_op[:, i]

    return examples_dict_op

Here is the code im using to train the model:

def train_and_eval():
"""Train and evaluate the model."""

m = build_estimator(model_dir)
m.fit(input_fn=lambda: input_fn(train_file_name, batch_size), steps=steps)

What would happen if I would call the fit function again with the same input_fn. Does it start at the beginning of the file again, or will it remember the line where it has stopped last time?

$\endgroup$
2

1 Answer 1

1
$\begingroup$

As there is no answer yet I want to try to give an at least somehow useful answer. Including the constants definitions would help a bit to understand the provided code.

Generally speaking a batch uses n times a record or item. How you define an item depends on your problem. In tensorflow the batch is encoded in the first dimension of a tensor. In your case with the csv file it might be line by line (reader=tf.TextLineReader). It could learn by column but I don't think that this is happening in your code. If you want train with your whole dataset (=one epoch) you can do so by using numBatches=numItems/batchSize.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.