1
$\begingroup$

I am trying to identify noisy intervals in geomagnetic data using logistic regression, working with scikit-learn.

Here is a typical spectrum of the data that I am working with: enter image description here

In this example, the data between 16:00 and 20:00 UTC -when the local railway system stops for the night- are assumed to be "clean" (1), while the remainder of the data are assumed to be "noisy" (0). This interval slightly changes from day to day, hence the need for a method allowing to automatically discriminate between clean and noisy data.

I train my model with 2 years of data. In order to get the same number of clean and noisy samples, for each day, I select 10 spectra between ~08:00 and ~09:40 as my noisy data and 10 spectra between ~17:00 and ~18:40 as my clean data. I therefore end up with a feature array X containing 14,600 samples (365x2x10x2):

for idx in np.arange(number_of_days):
               date=start_date+datetime.timedelta(int(idx))
               rd=RawData() 
               rd.populate(station_id,date,data_type, decimation_level)
               
               pxx,  freq, t, cax = plt.specgram(rd.decimated_data, Fs=decimated_frequency, detrend='mean', cmap='jet', scale='dB')              
               
               noisy_data=pxx[:,44:54]
               clean_data=pxx[:,94:104]                         

               if idx==0:
                    X_noisy=np.transpose(noisy_data)
                    X_clean=np.transpose(clean_data)
               else:
                    X_noisy=np.vstack((X_noisy,np.transpose(noisy_data)))
                    X_clean=np.vstack((X_clean,np.transpose(clean_data)))

 X=np.vstack((X_noisy,X_clean))
 y_noisy=np.zeros((X_noisy.shape[0]))
 y_clean=np.ones((X_clean.shape[0]))
 y=np.hstack((y_noisy,y_clean))

I then "whiten" my data and perform a 80/20 test-train split:

      X_scaled = preprocessing.scale(X)
      X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)

Finally, I fit my model and print out the score:

      clf = LogisticRegression()
      clf.fit(X_train, y_train)
      print("Model accuracy: {:.2f}%".format(clf.score(X_test, y_test)*100))

Alas, the score that I obatain is close to 99%, which as far as I understand means that I am overfitting my data. I know two remedies to overfitting:

1) Introduce regularization:

clf = LogisticRegression(C=regularization)

Regardless of what C value I select, I still end up being stuck with a score of ~99%.

2) Increase the size of the training dataset

If on each day I consider 12 clean spectra (from 08:00 to 10:00 UTC) and 12 noisy spectra (from 17:00 to 19:00 UTC), instead of the 10 originally selected, the score decreases from ~99% to ~68%. When considering this marginally larger dataset, the effect or regularization can be seen, with the score linearly increasing with the C value:

enter image description here

I should be pretty happy for nailing down this solution but I am surprised by how suddenly the score drops with a mere 20% increase in dataset size. Plus there is something fishy: if I consider slightly different intervals (say from 07:40 to 09:40UTC instead of from 08:00 to 10:00UTC), I revert back to a score of 99%. What am I doing wrong here?

$\endgroup$
2
  • $\begingroup$ Double check if your bigger dataset has no data leakage to training dataset. Try using k-fold cross-validation instead of just using a 80/20 split. $\endgroup$ Aug 10, 2020 at 20:44
  • $\begingroup$ Thanks for your reply, Pedro. I confirm that there is no data leakage from the bigger dataset to the training dataset. I will give k-fold cross-validation a shot! $\endgroup$
    – Sheldon
    Aug 10, 2020 at 21:02

1 Answer 1

0
$\begingroup$

If you are scoring 99% accuracy on the test dataset you are not overfitting, you have built a highly performant model. You are doing nothing wrong, the model has extremely high generalization to unseen data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.