I'm wondering if there is library support in python (such as sklearn) for doing KNN on a data set that has a custom distance matrix (positive definite) for each data point (x is a query point, $x_i$ is a data set point): $$ d(x,x_i) = (x-x_i)^TQ_i(x-x_i) $$
I know that for a fixed positive definite matrix for all data points, this is a metric that I can transform into $$ Q = A^TA \ \ \ \ \ \ \ d(x,x_i) = (Ax - Ax_i)^T(Ax - Ax_i) $$ Which I can compute via normal KNN by first transforming the input space via multiplying $A$.
My problem of having a separate matrix for each data point came up because I have a covariance around the neighbourhood of each point. KNN can then be interpreted as what are the most likely neighbourhoods this query point lies in. If a neighbourhood doesn't vary along a dimension then we should penalize difference along that dimension highly in terms of increasing distance.