Which desktop hardware is best for DL?

Question

I will be building my home Deep Learning workstation. Right now, I'm digging for some time about the best HW to use for home conditions.

The workstation will be used for my work as a developer, but I want to use it to learn and experiment with DL too.

Right now, got Quadro M1200 and T1000 4GB VRAMs on laptops (Precision 7520, 7550), but for training e.g. GPT2 with Czech Wikipedia, with 480 000 articles, it's pretty slow (260 hours on CPU for one epoch on 7550), https://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171.

I couldn't find any article/video on the internet, which can help me (because fast progressing in new HW too), so hope you can think with me here.

I went thru e.g. https://towardsdatascience.com/another-deep-learning-hardware-guide-73a4c35d3e86 which is old, https://pub.towardsai.net/best-workstations-for-deep-learning-data-science-and-machine-learning-ml-for-2021-4a6e43213b9e which is not talking about CPU cache or multi-GPU set, PCIE5, etc.

I tried some paid Google Colab, but still - it's so expensive, even with 100 credits, training in GPT2 is expensive as hell, so my own HW (for smaller models and optimization) is better. Setting up some Cloud Virtual GPU machine is the way I don't want to go, because of wasted time during configuring (I got rich experience with e.g webservers).

1. Don't want to use server CPUs (Threadripper, Xeons) - too expensive, not so universal (no pcie5, low frequency, etc.).

2. Already bought RTX 3060 12GB for the start, I was comparing it with ARC770 16GB, but CUDA seems to be faster and SW + drivers have still better support for nVidia. In the future, I will be adding a second GPU or will buy two RTX 3090/3090Ti - https://www.tensorflow.org/guide/distributed_training. I like the way CUDA works (been on GTC), but I don't like it's not open source e.g. oneAPI. For beginners, CUDA is the best path IMHO.

3. Will 192GB of RAM ok or 128GB is enough? In Colab, my "small" 480 000 articles GPT model ate almost 80GBs... but as I was reading articles above, that size of RAM == 2*VRAM is enough - this is not applicable in these times anymore or I just have not optimized code?

4. I was comparing i9-13900k with Ryzen 7950X3D. Ryzen has more PCI-E lines available, and it has 144MB of cache, so sometimes it can all be loaded into it, not into RAM - is it real world scenario, or Ryzen without 3D is enough?

5. Another CPU thing - threads - e.g. for IO operations is useless to have so many CPU cores and other calculations can be done on GPU or are good to have as powerful a CPU as 79xx/13900? Again, when comparing to my GPT2 experiment, almost the entire data preparation was done by CPU, the only final tf.fit is GPU...

6. Will be PCI-E 5.0 SSD for the actual dataset work ok, or it's overkill? (in May I'm expecting discs with 14GB/12GB read/write speed), again, when having e.g. 1M of files dataset, I'm finding it useful.

Answer 1

This question is complex because GPUs' power and memory are not linear.

I had some projects where some additional power or memory would have a much better result from one GPU to another, without any clear reason.

Furthermore, it highly depends on the data complexity and the data size.

It would have been great to have a GPU simulator website, at least to have a raw order of magnitude according to the main data sets, but I can't find any.

A good solution is to install a GPU cloud workbench to switch quickly from one GPU to another, test different scenarios, and eventually know which one is better.

It would be pretty cheap and 100% accurate.

Here are some good ones:

https://www.genesiscloud.com/

https://www.paperspace.com/

https://www.scaleway.com/

See also: https://lambdalabs.com/gpu-benchmarks

Answer 2

I built a PC specifically for ML last year and from my research at that time here are some of my conclusions. My datasets are quite a bit smaller than yours so I didn't spend too much time thinking about RAM/storage, but as for CPU/GPU:

1. Intel's MKL that is relevant for certain Python packages like NumPy seems to achieve appreciable performance improvements compared to using say OpenBLAS on an AMD CPU.
2. Support for AMD GPUs on certain ML packages like PyTorch is relatively recent, and as a result is probably less stable and less optimized than using an Nvidia GPU for the same task. Intel's ARC is even newer - I would say save yourself time and headache and grab an Nvidia GPU.

In the end I ended up getting an i5-12600K and RTX 3080 12G for these reasons. I don't think getting say, a 4090Ti will be worth the extra cost over a 4090 since your main concern is just getting as much VRAM as possible (bigger batches = better!). If you can really splurge the RTX A6000 has 48GB of VRAM but it might just be cheaper to get 2x 4090s and try to run them in parallel.