How does one fine-tune parameters and weights at the same time?

Question

I have been having my hands full with training a model to classify web pages. This is the first time ever that I am doing this, so I know very little about ML. I'm here to learn. :-)

Currently, I am using a Pipeline with FeatureUnion to run a a GridSearchCV to tune my parameters. However, I am slightly confused about how this process ought to go. For the whole process of creating a predictive model I read this answer on "Machine Learning Steps" question, which was helpful, but I still have some questions.

First of all: are parameters independent of your model? The answer linked above suggests first testing different models, and then do parameter tuning. However, I started with tuning my parameters on a Naive Bayes multinomial model, with the idea to test the models (e.g. svm, kNN, NBmn) afterwards with the 'optimal' parameters. But the answer above seems to suggest otherwise. Does this mean that the parameter tuning depends on the best model you choose, which makes it useless to first tune the parameters?

Apart from that, I am also not sure how to correctly tune my parameters per feature. I have 13 features, and put them all in the Pipeline. For things like TfidfVectorizer for eight features, I'd like to tune max_features, ngram_range, and max_df. However, because this takes quite a long time to run I thought I'd start with two features, make sure I get the optimal value, set them in the Pipeline (not in the parameter tuning), and add a feature. However, I feel that this is not the way at all. Is it required to the complete tuning as a single step, or is it possible to do it in steps?

Lastly, I noticed that you can also pass the feature weights of FeatureUnion as a parameter, and tune it. But as far as I have found you can only add specific values to test (example below) and not a range (e.g. 0-1 with steps of 0.1), which makes it hard to set the parameter values as you've got to guess which could be a good value. Is there a better way?

parameters = {
    'union__transformer_weights': [
        {'f1':1, 'f2':0.4, 'f3': 0.2,'f4': 1,'f5': 0.4,'f6': 0.1},
        {'f1':0.4, 'f2':0.9, 'f3': 0.2,'f4': 0.4,'f5': 0.4,'f6': 0.2},
        {'f1':0.8, 'f2':0.7, 'f3': 0.2,'f4': 0.7,'f5': 0.7,'f6': 0.4}
    ]
} 

Answer 1

Start each model with default parameters,use the evaluation metric to record error of each algorithm with default parameter. For majority of classification modeling AUC metrics gives the better results read here.

Not necessary that all features are important, do the feature selection. You can get the important features using random forest/xgboost algorithms, try to remove the least important features and check if evaluation score is improving. .

Now you think of tuning the parameters and yes each model have different parameters. Most common parameter is alpha(learning rate), to start with you can hand tuned alpha with values in range 0.0001 to 10 read here.

You can use more sophisticated methods to tuned parameters such Randomized and Grid search read here. Feature wights is way to tell your model that which is most/least important feature. You can get the idea of feature weights from the feature importance I explained at the top.

Answer 2

You can test each model with default parameters, and keep the model that seems best.

Or, for each model, you can tune the parameters for that model, to see what is the best that's achievable for each model (by separately tuning its parameters) -- then keep the best model. This takes more computation power but might be slightly better.

Yes, parameter tuning depends on the model. Each model may have a very different set of parameters, with entirely different meaning. You can't take parameters for a random forest and then try to use them as parameters for a SVM (for instance); that doesn't even make sense, as they have different parameters.

Yes, there are many possibilities on what order you do feature selection vs choosing a model vs parameter tuning.

I don't know what the "feature weights of FeatureUnion" are, so I can't answer that part.