There is a list of models and their performance and parameter count here. I prefer to use the MobileNet model initially. It is about as accurate as VGG but has about 4 million parameters versus the 140 million for VGG. Thus from a computational standpoint and thus training time it is far more efficient. If you modify it to use the callbacks ReduceLROnPlateau and ModelCheckpoint (documentation is here) you can in most cases achieve very good performance. I have found using the Adamax optimizer works very well, documentation for that is here. The code to use MobileNet is shown below
mobile = tf.keras.applications.mobilenet.MobileNet( include_top=False,
x=keras.layers.BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001 )(x)
predictions=Dense (len(classes), activation='softmax')(x)
model = Model(inputs=mobile.input, outputs=predictions)
for layer in model.layers:
model.compile(Adamax(lr=lr_rate), loss='categorical_crossentropy', metrics=['accuracy'])
If the model does not train well on your data set you can add more dense layers but it is always best to keep the model as simple as possible initially and try to optimize the hyper-parameters before making your model more complex.