I ran my job on a computing cluster: first with 1 node / 4 cores, then with 2 nodes / 32 cores. But the training time is pretty much exactly the same for both of them! 67 seconds per step.
I am trying to fine-tune GPT-2 for my text dataset (chat logs).
What can I do to get a performance increase corresponding to the increase in processing power with CPUs?