You can use any metric function that you specified when compiling the model.
Let's say you have the following metric function:
def my_metric(y_true, y_pred):
return some_metric_computation(y_true, y_pred)
The only requirement to this function is that it takes accepts the true y and the predicted y.
When you compile the model, you specify this metric, similarly to how you specify build in metrics like 'accuracy':
model.compile(metrics=['accuracy', my_metric], ...)
Notice that we are using the function name my_metric without ' ' (in contrast to the build in 'accuracy').
Then, if you define your EarlyStopping, just use the name of the function (this time with ' '):
EarlyStopping(monitor='my_metric', mode='min')
Make sure to specify the mode (min if lower is better, max if higher is better).
You can use it just like any build-in metric. This probably also works with other Callbacks like ModelCheckpoint (but I have not tested that). Internally, Keras just adds the new metric to the list of metrics available for this model using the function name.
If you specify data for validation in your model.fit(...), then you can also use it for EarlyStopping by using 'val_my_metric'.