As @shepan6 has mentioned, this could be because of your class balance in the validation set. You should print out the confusion matrix on both the training and the validation set. You might find that your algorithm gets two classes confused frequently. For example, with MNIST, 3 and 8 are often confused since they look similar when written. Using this example, if you don't have a lot of 3's and/or 8's in your validation set, then the accuracy would be higher in the validation set than the test set.

Even though a difference of 1% with 950 samples is just 9.5 samples, the change from 97% to 98% is probably not just do to randomness. The probability of a binomial with n = 950 and p = 0.97 being greater than or equal to 950*0.98=931 is about 3.7%, so statistically this is decent sized jump. (Here I'm ignoring randomness coming from uncertainty in the training set accuracy measurement, but with 3800 samples, there isn't much to worry about, especially since the validation error goes up in general.)

Of course, you may also want to review your validation code for bugs. Maybe you simply swapped your training and validation sets.

Lastly, you may have over-tuned your algorithm to do well on the validation set, and you may want to consider collecting new validation data if possible.

As @shepan6 has mentioned, this could be because of your class balance in the validation set. You should print out the confusion matrix on both the training and the validation set. You might find one high error class to be underrepresented in your validation set. You might also find that your algorithm gets two classes confused frequently. For example, with MNIST, 3 and 8 are often confused since they look similar when written. Using this example, if you don't have a lot of 3's and/or 8's in your validation set, then the accuracy would be higher in the validation set than the test set.

Even though a difference of 1% with 950 samples is just 9.5 samples, the change from 97% to 98% is probably not just do to randomness. The probability of a binomial with n = 950 and p = 0.97 being greater than or equal to 950*0.98=931 is about 3.7%, so statistically this is decent sized jump. (Here I'm ignoring randomness coming from uncertainty in the training set accuracy measurement, but with 3800 samples, there isn't much to worry about, especially since the validation error goes up in general.)

Of course, you may also want to review your validation code for bugs. Maybe you simply swapped your training and validation sets.

Lastly, you may have over-tuned your algorithm to do well on the validation set, and you may want to consider collecting new validation data if possible.

As @shepan6 has mentioned, this could be because of your class balance in the validation set. You should print out the confusion matrix on both the training and the validation set. You might find that your algorithm gets two classes confused frequently. For example, with MNIST, 3 and 8 are often confused since they look similar when written. Using this example, if you don't have a lot of 3's and/or 8's in your validation set, then the accuracy would be higher in the validation set than the test set. You could also find that one class is extremely rare in the training set and thus has high classification error, but over represented in the validation set.

Even though a difference of 1% with 950 samples is just 9.5 samples, the change from 97% to 98% is probably not just do to randomness. The probability of a binomial with n = 950 and p = 0.97 being greater than or equal to 950*0.98=931 is about 3.7%, so statistically this is decent sized jump. (Here I'm ignoring randomness coming from uncertainty in the training set accuracy measurement, but with 3800 samples, there isn't much to worry about, especially since the validation error goes up in general.)

Of course, you may also want to review your validation code for bugs. Maybe you simply swapped your training and validation sets.

Lastly, you may have over-tuned your algorithm to do well on the validation set, and you may want to consider collecting new validation data if possible.

As @shepan6 has mentioned, this could be because of your class balance in the validation set. You should print out the confusion matrix on both the training and the validation set. You might find that your algorithm gets two classes confused frequently. For example, with MNIST, 3 and 8 are often confused since they look similar when written. Using this example, if you don't have a lot of 3's and/or 8's in your validation set, then the accuracy would be higher in the validation set than the test set.

Even though a difference of 1% with 950 samples is just 9.5 samples, the change from 97% to 98% is probably not just do to randomness. The probability of a binomial with n = 950 and p = 0.97 being greater than or equal to 950*0.98=931 is about 3.7%, so statistically this is decent sized jump. (Here I'm ignoring randomness coming from uncertainty in the training set accuracy measurement, but with 3800 samples, there isn't much to worry about, especially since the validation error goes up in general.)

Of course, you may also want to review your validation code for bugs. Maybe you simply swapped your training and validation sets.

Lastly, you may have over-tuned your algorithm to do well on the validation set, and you may want to consider collecting new validation data if possible.