So I've had a rather "out there" idea. I want to train a dense network on a regression problem based on tabular data but I'd also like it to incorporate image data. My idea was to use a CNN based model for the images and have the output of that go directly into my dense network through a single linearly activated neuron. The other inputs to my dense model will be the different columns of the table data. It's roughly based on this tutorial.
Instead of coming up with some scoring system for the images, I'd like to train the single neuron at the end of my CNN at the same time as my DNN to perform this regression. My worry is that the DNN may reduce the weight of the image input before that neuron is able to learn an effective "score" for the images. This leads me to my question, would increasing the learning rate of the single neuron or decreasing the learning rate of the rest of my DNN work to mitigate that if it occurs?
I'll add that I'm going to perform some transfer learning on the CNN before it goes into the model to help "prime" it to be able to score these pictures but once it's incorporated into the DNN I plan on only training the single neuron output of it.