I would like to estimate the best eps value for the DBSCAN algorithm on this dataset by following this set of rules:
- Set a minPts: 10
- Compute the reachability distance of the 10-th nearest neighbour for each data-point.
- Sort the set of reachability distances and plot to get the elbow of the diagram (best eps value).
This is the first part of my code where I load the dataset:
import csv import sys import os from os.path import join from sklearn.cluster import DBSCAN from sklearn.neighbors import NearestNeighbors import matplotlib.pyplot as plt import numpy as np def load_data(file_path, file_name): with open(join(file_path, file_name)) as csv_file: data_file = csv.reader(csv_file,delimiter=',') temp1 = next(data_file) n_samples = int(temp1) print("n_samples=") print(n_samples) n_features = int(temp1) temp2 = next(data_file) feature_names = np.array(temp2[:n_features]) data_list = [iter for iter in data_file] data = np.asarray(data_list, dtype=np.float64) return(data,feature_names,n_samples,n_features) # --- Main program --- file_path="Datasets/" file_name3="CURE-complete.csv" data3,feature_names3,n_samples3,n_features3 = load_data(file_path, file_name3) fig = plt.figure(figsize=(8,8)) ax = fig.add_subplot(111) fig.subplots_adjust(top=1) ax.set_title('Dataset n. 3 of data points') ax.set_xlabel(feature_names3) ax.set_ylabel(feature_names3) plt.plot(data3[:,0], data3[:,1], '.', markersize=1.2, markeredgecolor = 'blue') plt.show()
This is where I compute the KNN-algorithm with ns (minpts) = 10.
ns = 10 #minpts nbrs = NearestNeighbors(n_neighbors=ns).fit(data3) distances, indices = nbrs.kneighbors(data3) distanceDec = sorted(distances[:,ns-1], reverse=True) plt.plot(list(range(1,len(distanceDec)+1)), distanceDec)
How can I improve my algorithm? As you can see in the following page, the plot (in my case it's reversed), should be different.
From what I understood the best eps value should be in the corner above 2.2, right?