I am trying to load two different keras models in parallel. I tried to use the functional API model:
input1 = Input(inputShapeOfModel1) input2 = Input(inputShapeOfModel2) output1 = model1(input1) output2 = model2(input2) parallelModel = Model([input1,input2], [output1,output2])
This works but it does not run in parallel actually. Inference time is just the sum of each model's individual inference time.
My question is should this run concurrently? I also tried to load them in different py files with gpu memory options. Still I haven't got parallelism (inference time is x1.5 for each model)
Is there any way to get inference time of both models as close to a single's model inference time? Is the only solution to add a second gpu?
UPDATE: in different scripts they seem to be able to run in parallel, so there must be a way to efficiently run in python/keras as well.