Once we know that the problem needs to be solved using supervised learning, how do we know if we have to use regression analysis or classification algorithm to solve it? Is there some thumb rules that can be used?
Most of the resources online give the standard example of prices of house for regression model and malignant/benign cancer plot for classification model. This isn't helping me much.
Is there come conceptual method to analyze the given problem?
A good rule of thumb is to look at the level of measurement of the target/response variable. If the response is measured on a nominal scale, the problem is a classification problem. Values on a nominal scale are for example labels of a categories where the categories have no natural order, like political parties in political science, species in biology, or parts-of-speech in grammar.
If the response is measured on a ratio or interval scale, you have a regression problem. Values on an interval scale are values where you can compare the degree of difference between values, but not the ratio between them, for instance temperature (on Farenheit or Celsius scales, but not Kelvin), or date values in a calendar. Values on ratio scales can be compared both with regards to degree of difference and ratio, like most physical quantities like mass, velocity or temperature on the Kelvin scale.
Ordinal scales are more difficult to place in either corner. I would generally say that you have a ranking problem with an ordinal response. However, the ranking problem can be approached using both classification, for instance using comparators, and and regression, like ordinal regression. Values on ordinal scales are ordered, or ranked, but you can't say anything meaningful about the degree of difference between any two values, for instance the ranking of racing drivers in a race.
Learning from a set of examples x mapping to y can be conceptualised as finding function f such that:
y = f(x)
x is vector of features, for e.g., car_model, car_version, city as vector of features for price prediction of used car.
y is output variable, for e.g., price of car x sold at.
if y is continuous, the problem is a regression problem
else if y is discrete, the problem is a classification problem
Continuous implies y can take any value [i, j] on Real scale and discrete implies y can take a value from set of {a, b, ..., d}
Normally the cases in which you use a Regression model is when you want to predict a continuous value from a set of given independent variables.
E.g : Let the following values be of the type [independent_variable, dependent_variable] or simply $[marks,height]$ and the values be $[2,0],[3,2],[4,5],[1,1]$. You fit a line or curve through these values ($[2,0],[3,2]$ etc.) and then see the case when a value of $[10,y]$ is given or marks of $10$ are obtained, what can be the $y$ (height) value from the fitted line or curve you had modeled.
Take a look at Linear Regression for the above type.
Classification model is used in the case when you got a set of independent variables as in the previous case but the dependent value used in training is not continuous value but tells what class the value belong to.
E.g : $([2,1],fail),([3,2],fail),([4,5],pass),([1,2],fail)$. Here [2,1] belongs to class $fail$ etc. So later time when a point say [7,8] is given, you will be finding which class (pass or fail) it might probably belong to.
For example SVM's for this case create a hyperplane(a multidimensional plane) and based on where the points falls in the space, it is going to find the class with some probability.
Simply, choose Regression if the dependent value is continuous else choose Classification if the dependent value is a class.
Are the target values ordered? Then it is likely to be regression.
Otherwise classification.
See Level of measurement.
Regression
Classification
However, I'd say for two classes it doesn't really make a big difference. In the case of predicting the gender of a user, you could also predict the probability of the user being female. It's just a minor variation of the same problem (ignoring this).
