Evaluation Metrics. For regression problems metrics like MSE or RMSE (is less sensitive to extreme values) are good defaults. For classification instead you can evaluate against accuracy if classes are balanced, otherwise look at the AUC of the ROC or PR (precision-recall) curves. In addition, the f1-score is also quite common, but in some other cases you may care more about errors and so the confusion matrix would give you an overview of the kind of error your model(s) made.
Basically, you pick one metric e.g. RMSE (for regression) and AUROC (AUC of ROC for classification), compute that for all your models and rank them accordingly. These metrics can be also used for selecting the best NN across training epochs (indeed you need to compute that on a validation-set.)
Compare and select models.
Since training one model (of one kind) gives you only a point estimate of its overall performance, which is an approximation, because training and test data are just limited. Moreover, there could be randomness in the model and/or training process that, at each run, may yield a different model with different performance.
Especially if you have not so many data, K-fold cross-validation allows you to estimate the bias and variance of your model quite easily. K-fold cross-validation allows you to estimate uncertainties related to the model and data. However, say your $k=10$ so would obtain $k$ models for each kind of them: you evaluate then on the metrics you care, and, basically, obtain a distribution of performance for each model class.
You should then aggregate the performance on your evaluation metric, obtaining average performance (e.g. by taking the mean) but also its standard deviation (i.e. variability in model predictions). For example, say model-1 achieves the best average but its std is quite large, while model-2 1% lower but the std is almost zero. So, what model do you choose?
When selecting the model you should consider both mean and std, or the overall distribution. To help yourself you can inspect a boxplot of the performance distribution of each class of models, such that you can visualize both average performance and the their associated variability.
In alternative is also possible to compute a $p$-value that provides you the probability that one class of models (e.g. SVM) is better than another (e.g. neural-nets).