Edit: Question has been edited for reopening (see comment section for justification)

Being to new text analytics, I haven't gotten the hang of navigating a typical workflow given the longer times associated with the larger feature sets. My question then is how does one navigate optimization decisions when the cost of exploring all are often so high?

To provide a specific example, in a single text analytics problem, I have prepared a few different transformations of my training set:

  • Stemmed vs lemmatized
  • Count vectorization vs TF-IDF vectorization
  • Full feature space vs 30% less features (identified by correlation analysis)
  • All cross-combinations of the above

In an effort to get a sense of which of the transformation sets above I should run further tuning on, I ran untuned RF, Logistic, Naive Bayes, SGD, and KNN models on (with cross validation). Unfortunately, it was clear no transformation combination really stood out as a likely "winner".

How does one proceed here? All decision points are of equal merit empirically, but exploring all seems strategically wasteful if not impractical.

  • $\begingroup$ Edited for the sake of reopening: I tried to specify a bit more of what I'm trying to get at here. I understand the question to touch on multiple points, but I can think of no other way to narrow it down. Moreover, I believe this an important, pragmatic example of a challenge many first-time nlp users face. $\endgroup$
    – Josh
    Oct 26 '20 at 23:38
  • 1
    $\begingroup$ For what it's worth I think the question is valid, but I can't think of any reasonably good answer. I'd be curious to know if anybody has one though! $\endgroup$
    – Erwan
    Oct 27 '20 at 0:05