Based on the paper you set k but want to sample greater than k and apply the log calculation.
I would recommend using the foreach function. Where you could say foreach() apply further reach than k.
where the following aglo from the paper could be interpreted as follows.
Algorithm 2 k-means||(k, ) initialization.
1: C ← sample a point uniformly at random from X
2: ψ ← φX(C)
3: for O(log ψ) times do
4: C 0 ← sample each point x ∈ X independently with probability px =·d2(x,C) φX(C)
5: C ← C ∪ C0
6: end for
7: For x ∈ C, set wx to be the number of points in X closer to x than any other point inC
8: Recluster the weighted points in C into k clusters
get points of k = 2
centroid is placed in middle for this example and
c1 = [(1,1),(1,2)]
This was achieved with the over sample of Euclid
foreach() point that satisfies the over sampling requirement.
see example here which uses for each:
# Cluster the data into two classes using PowerIterationClustering
model = PowerIterationClustering.train(similarities, 2, 10)
model.assignments().foreach(lambda x: print(str(x.id) + " -> " + str(x.cluster)))
So you will need to write the distance into the lambda.(if you provide code easier to help you).
K distance = k distance on perimeter which is the bottom red line in diagram
.foreach(lambda x: kdistance[get average] + then check prob(k prime) of k)
share some code and sample data please