2
$\begingroup$

DISCLAIMER: I understand this could be considered a subjective type of question, but I'm hoping it is specific enough to not be flagged. Please give it a chance.

I am doing a GridSearch in scikit-learn on a corpus of 50 books (all kinda average commercially published texts...each ~200 pages of text).

As I began playing with GridSearch, I got to thinking. Surely there must be some sort of 'best practices' for building a pipeline?

Sure, I can include and remove parameters, but I'm really just sorta stabbing around in the dark for the best score.

Possibly, it could make sense to create a smaller, exploratory pipeline using the larger, more effective parameters first. After running this basic pipeline, more parameters that have a smaller effect (in general) could be run next.

For example, say I'm GridSearching a TfidfTransformer. I am assuming that flipping the use_idf switch on or off is going to make a real impact on the scoring, whereas perhaps specifying alpha=0.000010 versus alpha=0.000015 won't.

In general, when using a pipeline, is there an alternative to "throw in the kitchen sink" that results in a more cognizant, informative (and possibly faster) experience?

$\endgroup$
2
+25
$\begingroup$

What you are asking for is a gradient based hyperparameter optimization. To enable such a grid search one would have to define a metric on the space of all hyper parameter combinations for the current pipeline.

I don't think this makes sense in your example because of the binary nature of the use_idf flag. Lets say your parameter combinations are (use_idf, alpha). Then you would have to do decide what the distance between (True, 0.00015) and (True, 0.00010) is. In your case it should be smaller than the distance between (True, 0.00015)and (False, 0.00015). So

d((True, 0.00015), (True, 0.00010)) < d((True, 0.00015), (False, 0.00015))

But what about the distance between (True, 0.00015) and (True, 0.05)? Will it be still smaller than d((True, 0.00015), (False, 0.00015))? I think you get the point.

By choosing the metric you will subjectively influence the search for the best hyper parameters. Probably a random search would be more suitable as suggested in the following paper

@article{bergstra2012random,
  title={Random search for hyper-parameter optimization},
  author={Bergstra, James and Bengio, Yoshua},
  journal={The Journal of Machine Learning Research},
  volume={13},
  number={1},
  pages={281--305},
  year={2012},
  publisher={JMLR. org}
}
| improve this answer | |
$\endgroup$
0
$\begingroup$

auto-sklearn team is working on finding a way to fully automate the pipeline. It has delivered good results for me so far. Better as what I have got manually with the GridsearchCV. This can be an answer to your question if you 're happy to use such tool or read their source code.

And at ML4AAD you can find further links to the attempts to solve this problem.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.