I've done a simple naive bayes classification task with a very small data set. As the training set size increases from 100 data points to 300 data points, the F1 score on the test set decreases. But why? Is it likely overfitting? What are the first places I should examine to explain this behavior?
Overfitted model is something that shows very less error on your training set and then when you test it on a test set, it fails badly. This might be due to class imbalance and you might be giving something that it has not seen or due to using more number of features than you were supposed to, making the model to fit in every datum. Wiki page is nice overview read for this.
Answering your next question, F1 score as might already be knowing has two components in it, Precision and Recall. Check whether there are reasonable number of TP,TN when compared to FN, FP and if you find those numbers to be fair enough, then F1-score is just a metric for you. There is no benchmark score for an awesome F1 score. The decrease might be just because your model has just more data to predict now.
So overfitting can lead to less F1-score but reduced F1-score does not mean overfitting always.