I believe current implementations of the best known libraries will allow you to implement any network you come across. So it boils down to the aspects of taste and practicality.
As far as deciding on a framework based on the network you want to create; if the network has branches or separate control flows (data transfer or logical) that are used conditionally during training, PyTorch is likely more useful. Otherwise, the best starting point is likely to use Keras - a high-level API over Tensorflow.
The (very) general rule of thumb is:
Use Pytorch for experimentation, and Tensorflow for production
Now the caveats to the above sweeping statement:
- Both libraries are making massive efforts to cover "the other" library's strengths:
The latest release of Pytorch (v1.0) offers several methods of making production ready code - so ways of compiling it to improve performance as well as the use of 16-bit floating point operations (as opposed to the standard 32-bit). They have also started rolling out a C++ API. These are all steps to make full-scale products for deployment. Tensorflow has introduced its eager API, for eager execution, meaning you can essentially build dynamic graphs, just as in PyTorch. The enables faster and more interactive prototyping.
- The experience you have while developing might be more important:
If you are a pure Python developer, PyTorch will feel more natural. If you have been a general software engineer in other statically types languages (such as C++ and Java), Tensorflow may feel more logical.
Overall, Tensorflow along with (the now built-in) Keras wrapper, is a more mature library with a larger community and longer history, meaning more support online. PyTorch has a higher level of excitemnet around it, as far as I can tell. It is used very actively in research and newer open-source projects/courses, which indicates it may become a dominant force.