Where is the majority of Tensorflow memory usage?

Trying to wrap my head around where Tensorflow starts to use a lot of memory.

If I have a batch_size=10 and my samples are 150MB each, then that’s only ~1GB at a time or less if the samples are processed 1 at a time.

Is it more about the accruing size of the neural net itself as I go through more epochs? If I try to train against 10,000 of these 150MB samples, am I more likely to run out of memory toward the end of the training?

The total memory requirement will be a product of the size of the model (number of parameters) along with the batch size. Each single sample that gets sent through the network will effectively influence each of the $$N$$ weights. A batch of size $$B$$ will mean that there needs to be roughly $$N \times B$$ memory available.
If you model is already fixed, you can start with a large batch size, decreasing it slowly until you no longer receive an outOfMemory error. You could also try computing exactly what your memory requirement are, but there are some other factors that might make that a little difficult, so it is faster to try the first approach.
If you are using a GPU, you ca look at the output of the terminal command nvidia-smi, you can see the available memory of the GPUs. You will notice it essentially becomes all used as soon as training begins. This is because Tensorflow, by default, will occupy all available memory. There are ways around that, so search for allow_growth in the Tensorflow docs for relevant information and configuration tactics.