# Accuracy over different sample sizes from dataset

What I'm trying to do is predict how much more data would help in a classification task.

So, what I'm doing is bootstrapping entries in my dataset to get a sample, with a specified size. Then, I fine-tune a KNN model on the sample, and compute it's accuracy. I do this multiple times for one specified size, storing the accuracies in a list, from which I can compute the mean accuracy as well as the standard deviation of accuracies. And then rinse-and-repeat, for a different specified number of samples.

After doing this for enough different sample sizes, I practically get a new dataset, which I can fit a line to and predict how much more data would help.

However, this new "Accuracy vs. Sample Size" dataset surprises me.
What I expected was that at small sample sizes, more data would help out a lot, but with bigger sample sizes, more data would help out less. I'm also interested in the random noise between points, I understand that there will generally be more noise with smaller sample sizes, but the change is just so sudden (at around the sample_size=100 point). These results stayed consistent throughout multiple simulations. Can anyone explain this behaviour to me? I have a feeling it might be to do with how ridiculously small my dataset is, and perhaps because of how KNNs work. But I can't say for certain, or why it would be the case.

Here's a minimal working example (in python):

# Imports
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV

import warnings
warnings.filterwarnings("ignore")

def auto_knn(X, y):
"""
Returns accuracy of KNN with fine-tuned 'n_neighbours', fitted on 'X' and 'y'.
"""
knn = KNeighborsClassifier()
min_range, max_range = 6, 9
found_best = False
knn_search = GridSearchCV(knn, [{
"n_neighbors": list(range(min_range, max_range))
}], cv=5, scoring="accuracy", return_train_score=True)
knn_search.fit(X, y)
if knn_search.best_params_["n_neighbors"] == min_range and min_range != 1:
min_range, max_range = min_range-1, max_range-1
elif knn_search.best_params_["n_neighbors"] == max_range-1:
min_range, max_range = min_range+1, max_range+1
else:
found_best = True
return knn_search.best_score_

def knn_scores_from_subsample(X, y, subsample_size, num_iters):
"""
Samples 'X' and 'y' with replacement to get new dataset of specific size, then fine-tunes KNN and retrieves accuracy. Resamples and retrieves accuracy 'num_iters' times. Returns list of accuracies.
"""
accuracys = []
for i in range(num_iters):
print("\r["+"".join(["=" for _ in range(round(i/num_iters*100))])+"".join([" " for _ in range(round(100-i/num_iters*100))])+"] - "+str(round((i/num_iters)*100, 1))+"%", end="")
selected_indexes = np.random.randint(0, len(X)-1, (subsample_size,))
accuracys.append(auto_knn(X.iloc[selected_indexes], y.iloc[selected_indexes]))
return accuracys

import requests
from io import StringIO

# Here, I'm using the Heart Failure Dataset (https://www.kaggle.com/andrewmvd/heart-failure-clinical-data),
# I've already done feature engineering, selection and preparation, so I've uploaded the prepared dataset,
# to easily retrieve here.
file_id = orig_url.split('/')[-2]
url = requests.get(dwn_url).text
csv_raw = StringIO(url)
data = data.drop("Unnamed: 0", axis=1)
X = data.drop("DEATH_EVENT", axis=1)
y = data[["DEATH_EVENT"]]

# Generate 'accuracy' vs. 'specific sample size'
# NOTE: This can take a very long time, to generate the image (above), it took many hours.
scores_for_n = {}
for num_points in range(20, 300, 10):
scores_for_n[num_points] = knn_scores_from_subsample(X, y, num_points, 3500) # This will resample (with replacement) the dataset 3500 different times (and compute the accuracy).
print("\nFinished "+str(num_points)+" points. Got a mean accuracy of "+str(round(np.array(scores_for_n[num_points]).mean()*100, 3))+"% With a standard deviation of: "+str(round( np.array(scores_for_n[num_points]).std()*100 ,3))+"%")

# Plot 'accuracy' vs. 'specific sample size'
plt.scatter(data_points, accuracies)
plt.xlabel("Dataset Size")
plt.ylabel("Mean Accuracy")


Also, I've uploaded my "Accuracy vs. Sample Size" dataset to Google Drive, so you can retrieve it for your own inspection, here's the code to do it:

import pandas as pd

import requests
from io import StringIO

file_id = orig_url.split('/')[-2]
url = requests.get(dwn_url).text
csv_raw = StringIO(url)
data = data.drop("Unnamed: 0", axis=1)

• The accuracy reported is on a validation set or in the same sample you trained your model on? – Julio Jesus Feb 16 at 16:08
• It's evaluated using cross-validation, with 5 folds. – MartinM Feb 16 at 16:10