I think what @Kevin H has said is right.
There are couple of more thing which I would suggest you to spend more time on
- Spend more time on understanding data, what does each word in each scenario means.
- Generate a Stopword Dictionary to remove all the unnecessary words, Lemmatization of words.
- Based on your Business Understanding make sure that all the important details are capture.
- Using a Single Model helps in generalizing the results but before deciding that as the final model, make different models for different business problems and do ensemble of them and validate to know which model give better results(Ensemble or Single Model).
Main reason for suggesting Ensemble is Navie Bayes classifier works best with short sentence but for rest that is not the best model.
Regarding Sampling, I think to do that you need to understand data clearly as to extract sample you need to make sure that every dimension of the population is covered(Sample should exactly represent the population). So, you should be very careful when implementing Sampling techniques.
Choose the best model, you need to try all the methods apply them in all applicable conditions and based on your Business Application and results achieved, you need to decide but generally people implement Ensemble to achieve best results(in most of the cases).
If you need more on Ensemble or any other technique let me know, will help you!