Suppose I have trained a multi variate linear regression model on a particular training set, and the model parameters $\theta=[\theta_1,\theta_2,\ldots, \theta_n]$ were determined by minimizing a cost function, say MSE (Mean Squared Error).
Now, if for some reason I decide to change my choice of error function from MSE to MAE (Mean Absolute Error), will this change my model parameter $\theta$ or will it remain the same?
Yes, it will impact because when you change the loss function, the numerical value of the loss function will change. So, this will change gradient values of the parameters during the back propagation. Therefore, change in loss function will impact the parameters.
If you change the objective function, the optimal solution to the objective function is likely to change though sometimes they might ended up to be the same.
If the optimal solution remains the same, we wouldn't have to have differen objective function. MSE pays more attention to large error and MAE tend to give sparser solution.
