I will be building my home Deep Learning workstation. Right now, I'm digging for some time about the best HW to use for home conditions.
The workstation will be used for my work as a developer, but I want to use it to learn and experiment with DL too.
Right now, got Quadro M1200 and T1000 4GB VRAMs on laptops (Precision 7520, 7550), but for training e.g. GPT2 with Czech Wikipedia, with 480 000 articles, it's pretty slow (260 hours on CPU for one epoch on 7550), https://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171.
I couldn't find any article/video on the internet, which can help me (because fast progressing in new HW too), so hope you can think with me here.
I went thru e.g. https://towardsdatascience.com/another-deep-learning-hardware-guide-73a4c35d3e86 which is old, https://pub.towardsai.net/best-workstations-for-deep-learning-data-science-and-machine-learning-ml-for-2021-4a6e43213b9e which is not talking about CPU cache or multi-GPU set, PCIE5, etc.
I tried some paid Google Colab, but still - it's so expensive, even with 100 credits, training in GPT2 is expensive as hell, so my own HW (for smaller models and optimization) is better. Setting up some Cloud Virtual GPU machine is the way I don't want to go, because of wasted time during configuring (I got rich experience with e.g webservers).
Don't want to use server CPUs (Threadripper, Xeons) - too expensive, not so universal (no pcie5, low frequency, etc.).
Already bought RTX 3060 12GB for the start, I was comparing it with ARC770 16GB, but CUDA seems to be faster and SW + drivers have still better support for nVidia. In the future, I will be adding a second GPU or will buy two RTX 3090/3090Ti - https://www.tensorflow.org/guide/distributed_training. I like the way CUDA works (been on GTC), but I don't like it's not open source e.g. oneAPI. For beginners, CUDA is the best path IMHO.
Will 192GB of RAM ok or 128GB is enough? In Colab, my "small" 480 000 articles GPT model ate almost 80GBs... but as I was reading articles above, that size of RAM == 2*VRAM is enough - this is not applicable in these times anymore or I just have not optimized code?
I was comparing i9-13900k with Ryzen 7950X3D. Ryzen has more PCI-E lines available, and it has 144MB of cache, so sometimes it can all be loaded into it, not into RAM - is it real world scenario, or Ryzen without 3D is enough?
Another CPU thing - threads - e.g. for IO operations is useless to have so many CPU cores and other calculations can be done on GPU or are good to have as powerful a CPU as 79xx/13900? Again, when comparing to my GPT2 experiment, almost the entire data preparation was done by CPU, the only final tf.fit is GPU...
Will be PCI-E 5.0 SSD for the actual dataset work ok, or it's overkill? (in May I'm expecting discs with 14GB/12GB read/write speed), again, when having e.g. 1M of files dataset, I'm finding it useful.