I developed a CNN based on EfficientNet in order to predict the weight of piles of some materials in an image (the labels are the weights in kg and the input is RGBD tensors of the object). I have two different materials (m1 and m2) which are similar but can be classified easily (you can see the difference between them). The specific weight of each material is different (m1 is 3 times heavier than m2). I trained my network using data of m1 only and got pretty good results: MAE of 0.12 kg per pile (who weights 0-6 kg). I used 30K tensors (24k for train and validation and 6k for test).
Now I want to take the trained network and transfer it to predict the weights of m2. The task is the same and the images are similar (were taken in the same environment) so the data is similar. I was wondering what would be the best way of transferring this model (and weights learnt by the model trained on m1) to predict the weights of the other material (m2).
So my question is: What will be the best way to transfer the model trained using data of m1 to predict data of m2 (which is similar but the specific weight is much lower)?
Should I use the weights learnt using m1 as initializers and retrain the whole model to predict m2? should I freeze most of the layers and retrain the last few layers only? How much tensors of m2 should I use? My m2 dataset also includes 30K images, should I use all of them or just a part of them?
Do you have any ideas I didn't mention?
Thanks in advance!!