Edit: Question has been edited for reopening (see comment section for justification)
Being to new text analytics, I haven't gotten the hang of navigating a typical workflow given the longer times associated with the larger feature sets. My question then is how does one navigate optimization decisions when the cost of exploring all are often so high?
To provide a specific example, in a single text analytics problem, I have prepared a few different transformations of my training set:
- Stemmed vs lemmatized
- Count vectorization vs TF-IDF vectorization
- Full feature space vs 30% less features (identified by correlation analysis)
- All cross-combinations of the above
In an effort to get a sense of which of the transformation sets above I should run further tuning on, I ran untuned RF, Logistic, Naive Bayes, SGD, and KNN models on (with cross validation). Unfortunately, it was clear no transformation combination really stood out as a likely "winner".
How does one proceed here? All decision points are of equal merit empirically, but exploring all seems strategically wasteful if not impractical.