It is not possible to predict outcomes of ideas like this with certainty. Too much depends on the specifics of your problem and dataset. You will have to do the experiments yourself. With a decent auto-differentiation engine like TensorFlow or PyTorch, you could try all your different ideas really quickly, with a few variations.
Here is what I think will happen from an intuitive guess:
Should I try to predict y1, y2, y3, then calculate Y and pass it to a loss function
I think this has potential to be the strongest model, as it incorporates knowledge that you have of the problem into the structure of your model, whilst not adding more hyperparameters. This may also allow you to use less parameters to achieve the same accuracy, plus may work as a form of regularisation.
Should I try to predict Y standalone and ignore y1, y2 and y3?
This should be considered the default/benchmark model, and you will want to run that anyway. If you only make one model, make this one.
If you run comparisons, you will want to compare best model of this type with best model(s) of other types. Don't just try the same network with an altered top layer to change the prediction mechanism - that will not explore your idea well enough to give you an answer.
Should I try to predict the 4 of them and penalize my model when Y != y1 * y2 * y3 ?
I expect this would be the hardest to train. Although it may still add some regularisation effect, your problem is that the "best" model in this category may rate consistency over accuracy - you will need to play with different weightings of the penalty metric to find a compromise loss function.
Remember to use cross-validation and test data in order to search parameters and get fair unbiased measures of performance of the best model from each of the approaches at the end.
