Machine Learning - Choice of features for determining hypothesis

Question

I am a newbie to machine learning and have the following elementary questions.

1. Given a labeled dataset with multiple features, is it for a ML algorithm determine on its own what features are to be considered for determining the hypothesis function? Or is it for a user who is training the algorithm to explicitly indicate what features are to be considered?

2. Is there a mapping between problem space versus the hypothesis function type? As an example based on past experience, can we confidently state that for a housing price prediction problem, linear equation is appropriate. While for determining diabetes based on a given dataset, a quadratic hypothesis equation has proven reasonable.

Or is it usual for a designer to specify a custom function and allow the ML algo to determine the coefficients of this custom function (e.g. square root of sum of squares of 9 of 15 features divided by e^n) by reducing the error using gradient descent?

Answer 1

1. There are algorithms that use Feature Selection to utilize only the "best" features for the given dataset (variants of adaboost for example). There are also algorithms randomly use subsets of the features (such as random forest). Finally there are algorithms that learn weigh your inputs and use them accordingly (Neural networks and the like)
2. There is "common knowledge", but it is more related to the type of features and classification problem that you are facing (i.e. regression vs. classification, continuous vs discrete, binary vs. multi-class, etc.).

If you have a specific problem that you can model using such a function then it is a completely reasonable approach to design such a function. Once you have modeled your problem with such a function and defined your loss function - you could use any optimization method that seems reasonable. However it is beneficial to try and find similar problems that others have solved before and understand how their solution relates to your problem.

Answer 2

Firstly, I would like to tell you that domain knowledge plays a crucial role applying machine learning models.

Coming to your question 1: ML algorithm will not strictly determine weather to use a feature or not. ML algorithm will only tell you about the relevance of a particular feature present in your feature set. For example, while doing regression, variables associated with higher coefficients are of greater importance than the ones having lower weights,it might be very close to zero as well, which gives you hint to not consider the feature. Similar is the case with the fbeta score of a variable while we deal with trees. User can specify what features are to be considered. User could also make some derivative features which helps a lot while dealing with ML problems Generally this is an iterative process.

For the second question : There is no specific mapping between the problem statement and hypothesis. You may develop some of them with experience.

You should not(usually) choose such custom functions as they might be fitting your training data perfectly, but could give poor results on test data as there is loss of generality by using a specific function. You could try different algorithms, with making new features, using ensemble methods and other machine learning techniques to improve your score

Answer 3

1. Choosing the best set of features is the objective of feature engineering. Feature engineering combines domain expertise and algorithms in order to extract the best set of predictors for a given problem. The process follows these steps:

   • Domain experts determine which factors may be useful towards predicting the target variable.
   • These factors are transformed into input features
   • Data is collected
   • Feature selection is applied in order to measure the relative importance of each feature to predict the target variable. We can both measure such importance for each feature individually or for combinations of features. Multiple feature selection methods exist, so you would have to choose which one is the best fit for your project requirements.
   • Additionally, and as part of the feature selection process, one could create new features by combining sets of raw input features.
   • We check how the features work with the model, and repeat the process from step 1 if necessary by contacting the domain experts again.

As you can see, creating the best set of features is not an automatic process, and domain expertise is very important. This is a very common thing in many data science related tasks, as you will see in the next point.

2. There is not such a thing that a perfect mapping between type of problems and algorithms. The model you may end up using will strongly depend on the nature of your data. You would need to explore the data and ask the domain experts in order to make an informed decision on what method to use. Experience will make the process easier with time, but it is likely that you will have to apply different types of models to your problem in order to look for the best one.