Escaping from overfitting hell: introducing regularization vs increasing training data

Question

I am trying to identify noisy intervals in geomagnetic data using logistic regression, working with scikit-learn.

Here is a typical spectrum of the data that I am working with:

In this example, the data between 16:00 and 20:00 UTC -when the local railway system stops for the night- are assumed to be "clean" (1), while the remainder of the data are assumed to be "noisy" (0). This interval slightly changes from day to day, hence the need for a method allowing to automatically discriminate between clean and noisy data.

I train my model with 2 years of data. In order to get the same number of clean and noisy samples, for each day, I select 10 spectra between ~08:00 and ~09:40 as my noisy data and 10 spectra between ~17:00 and ~18:40 as my clean data. I therefore end up with a feature array X containing 14,600 samples (365x2x10x2):

for idx in np.arange(number_of_days):
               date=start_date+datetime.timedelta(int(idx))
               rd=RawData() 
               rd.populate(station_id,date,data_type, decimation_level)
               
               pxx,  freq, t, cax = plt.specgram(rd.decimated_data, Fs=decimated_frequency, detrend='mean', cmap='jet', scale='dB')              
               
               noisy_data=pxx[:,44:54]
               clean_data=pxx[:,94:104]                         

               if idx==0:
                    X_noisy=np.transpose(noisy_data)
                    X_clean=np.transpose(clean_data)
               else:
                    X_noisy=np.vstack((X_noisy,np.transpose(noisy_data)))
                    X_clean=np.vstack((X_clean,np.transpose(clean_data)))

 X=np.vstack((X_noisy,X_clean))
 y_noisy=np.zeros((X_noisy.shape[0]))
 y_clean=np.ones((X_clean.shape[0]))
 y=np.hstack((y_noisy,y_clean))

I then "whiten" my data and perform a 80/20 test-train split:

      X_scaled = preprocessing.scale(X)
      X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)

Finally, I fit my model and print out the score:

      clf = LogisticRegression()
      clf.fit(X_train, y_train)
      print("Model accuracy: {:.2f}%".format(clf.score(X_test, y_test)*100))

Alas, the score that I obatain is close to 99%, which as far as I understand means that I am overfitting my data. I know two remedies to overfitting:

1) Introduce regularization:

clf = LogisticRegression(C=regularization)

Regardless of what C value I select, I still end up being stuck with a score of ~99%.

2) Increase the size of the training dataset

If on each day I consider 12 clean spectra (from 08:00 to 10:00 UTC) and 12 noisy spectra (from 17:00 to 19:00 UTC), instead of the 10 originally selected, the score decreases from ~99% to ~68%. When considering this marginally larger dataset, the effect or regularization can be seen, with the score linearly increasing with the C value:

I should be pretty happy for nailing down this solution but I am surprised by how suddenly the score drops with a mere 20% increase in dataset size. Plus there is something fishy: if I consider slightly different intervals (say from 07:40 to 09:40UTC instead of from 08:00 to 10:00UTC), I revert back to a score of 99%. What am I doing wrong here?

Answer 1

If you are scoring 99% accuracy on the test dataset you are not overfitting, you have built a highly performant model. You are doing nothing wrong, the model has extremely high generalization to unseen data.