Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am reading the book "Data Science for Business" by Foster Provost & Tom Fawcett. Only a fourth of the way through.
I am unclear about the concept of Objective Function. I will nevertheless take a stab and would like to seek feedback.
Say that we decided to use standard linear regression for a given problem (assume we have training data). Is the goal of objective function to determine the best parameters (weights) for this training data?
Thanks in advance!
Objective function is a function that needs to be optimized (either maximized or minimized) given certain constraints.
In linear regression, you have training data of m samples and n features, and the model: y = w0x0 + w1x1+ ... +wnxn.
The goal is searching for the optimum value of w0, w1, ...wn so that you expect for given unseen data X you can accurately predict the value of y. To reach the goal, you train the model on m samples. Start with n random weights, doing iteration to update the weights.
During the iteration, you are doing evaluation to the weights. How? By measuring how close the value of y obtained from current weights to the actual value of y. You may pick any formula to define "how close", the most popular is the Mean Squared Error (MSE).
Then the objective is to minimize the MSE function, since smaller error means that y obtained from current weights are close enough to the actual y.
So I can say that the goal of the objective function is optimizing weights by minimizing the error.
The objective function is the mathematical equation which combines the parameters in a useful sense. For linear regression this is a simple linear combination of weighted features.
You will need to choose which objective function is best suited for your problem. Then based on the function you selected you will know what are the model parameters that will need to be fit using an appropriate training method.
One of the skills of the data scientist is to identify the objective function which has the best likelihood of success given a problem statement.
