As @Anony-Mousse pointed it, on DBSCAN index structures are often used in order to decrease execution times. K-d-trees are one example but this one works well just in small dimensions.
You had right intuition about what slowing down, the computation of every distance from all point to one is O(n) time complexity but applied to every points it becomes O(n2) which is something we desire to avoid on important set of points.
Moreover you may reconsider your clustering approach because DBSCAN is known to works well with small dimensional datasets due to hyperball volume fast decrease with higher dimensions.
Keep trying implementing algorithms by yourself it is best way to learn even if existing mature library do often a better job but are also more difficult to understand, one step after another you will be able to write better algorithms and contribute to open source libraries.