0
$\begingroup$

I have a regular 2D grid of data points (X, Y) with each point having a value.

I'd like to identify clusters and then anomalies that don't belong to those clusters.

I'm trying to understand the best way to do this and it looks like DBSCAN might be an appropriate algorithm to use but I'm a bit confused.

I'm not clear whether I should collapse the 2D (X,Y) data down into 1 dimension and run DBSCAN on the 1D array. That would lose the spatial component of the data but would then identify clusters based on the value of the data point.

Or do I run DBSCAN on a 3D array (the X index, the Y index and it's corresponding data value for that X,Y point)? That sounds intuitively like it might be the better way to approach this but I'm not 100% sure.

If anyone could point me towards an example of using DBSCAN for spatial clustering that'd be really helpful.

$\endgroup$
1
$\begingroup$

Clustering in X,Y,Value

Clustering in 3d is great. But be careful with feature scaling in this case. Presumably X, Y have the same scales - so unless you want to treat the different directions differently, make sure not not apply any normalization. As that would distort the grid.

Your Value column on the order hand, might be on a very different scale from the X,Y values. If not corrected for this creates an implicit and usually quite arbitrary feature weighting, which is rarely desirable. If you want to weight the Value column equally to X and Y, then scale the Values by the either mean,std or max,min of the X/Y columns.

If you do want to weight the individual attributes differently, then you can introduce additionally some weighting factors that scales them.

PS: remember that DBscan hyperparamters, such as epsilon is also given in the same unit as your feature space.

Anomalies in Value given X,Y location

Often an interesting question is "is this value anomalous compared to the normal for this location?". This can be done on X and Y axis individually, but often there are interactions between X,Y and one should consider the two dimensions together. Here are some approaches for that:

Binning the 2D (X,Y) space

Divide the X and Y axis into a grid. Typically with uniformly sized cells. Assign each grid cell an location identifier. Then use this identifier as a feature.

This approach is simple and with a lot of dense data it usually works pretty OK. But the grid can fit the data pretty poorly, datapoints very close to eachother end up in different cells, and then quite distant in feature space.

Clustering in the 2D (X,Y) space

An alternative location scheme is to use clustering. Cluster the values just based on X,Y - and then use the cluster number as the location identifier. You may also want to include distance from the cluster center as a feature - since some values might fall pretty far from a cluster. Angle from the cluster can in some cases also be informative.

This has a much better shot at creating good location features for the data. But is more complex. One also should consider when one wants to re-cluster.

Aside

Make sure to plot your data and look at it! For example use a 2d-plot with X,Y as coordinates, and the values using color. You may also want to do the same kind of plot, but then to use Anomaly Score as the color. If you upload a plot here, it will also be much easier to give good advice.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the response. That's exactly what I was looking for. I currently have the data binned onto a regular grid and I'm using X, Y and Z as the features. It seems to work ok but the trade-off in scaling the features is the biggest issuem, as you say. At the minute I'm just standardising both the latitude and longitude and the data value and that seems to work if I play with the eps value. $\endgroup$ – Rob Sep 6 at 12:06
  • $\begingroup$ No prob. If you have lat/lon, then I recommend not standardizing. 1) It lets you set eps in a meaningful unit - actual distances in lat/lon space. 2) It wont break your model when the range of your data (and thus your standardization parameters) changes. $\endgroup$ – jonnor Sep 6 at 14:46
  • $\begingroup$ I dont understand why you bin, but then use X,Y as the features though... $\endgroup$ – jonnor Sep 6 at 14:47
  • $\begingroup$ Maybe I'm still misunderstanding but if the data is binned and the only feature you pass it is the list of Z values, it doesn't know anything about one data point being next to another. Doesn't the position need to somehow be a feature (i.e. by passing it the lat/lon location)? In response to your other comment, I still normalise as the data values are on very different scales to the lat/lon. I might have 0.1 degree lat/lon bins but the data values can range from 0 to 4000. It sounds like I'm still not quite getting something! $\endgroup$ – Rob Sep 7 at 15:05
0
$\begingroup$

Check out this comparison here. Intuitively I would say fit on everything you've got, don't throw anything away.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.