1
$\begingroup$

I have an CNN object detection model which has two heads(outputs) with tensor names 'classification' and 'regression'.

I want to define a metric function that accepts both the outputs at the same time, so that it looks into the regression predictions to decide which indexes to retain and use those indexes to select tensors from classification predictions and calculate some metric.

my current metric function defined with help from this link:

from tensorflow.python.keras.metrics import MeanMetricWrapper

class Accuracy2(MeanMetricWrapper):

    def __init__(self, name='dummyAccuracy', dtype=None):
        super(Accuracy2, self).__init__(metric_calculator_func, name, dtype=dtype)
        self.true_positives = self.add_weight(name='lol', initializer='zeros')

    @classmethod
    def from_config(cls, config):
        if 'fn' in config:
          config.pop('fn')
        return super(Accuracy2, cls).from_config(config)


    def update_state(self, y_true, y_pred, sample_weight=None):
      print("==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@===")
      print("Y-True {}".format(y_true))
      print("Y-Pred {}".format(y_pred))
      print("==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@===")

      update_ops = [self.true_positives.assign_add(1.0)]
      return tf.group(update_ops)

    def result(self):
      return self.true_positives

    def reset_states(self):
      # The state of the metric will be reset at the start of each epoch.
      self.true_positives.assign(0.)

which I call the during model compilation as :

training_model.compile(
    loss={
        'regression'    : regression_loss(),
        'classification': classification_loss()
    },
    optimizer=keras.optimizers.Adam(lr=lr, clipnorm=0.001),
    metrics=[Accuracy2()]
)

screen log during tf.estimator.train_and_evaluate is:

INFO:tensorflow:loss = 0.0075738616, step = 31 (11.941 sec)

INFO:tensorflow:global_step/sec: 4.51218

INFO:tensorflow:loss = 0.01015341, step = 36 (1.108 sec)

INFO:tensorflow:Saving checkpoints for 40 into /tmp/tmpcla2n3gy/model.ckpt.

INFO:tensorflow:Calling model_fn. ==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@=== Tensor("IteratorGetNext:1", shape=(?, 120087, 5), dtype=float32, device=/device:CPU:0) Tensor("regression/concat:0", shape=(?, ?, 4), dtype=float32) ==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@=== ==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@=== Tensor("IteratorGetNext:2", shape=(?, 120087, 2), dtype=float32, device=/device:CPU:0) Tensor("classification/concat:0", shape=(?, ?, 1), dtype=float32) ==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@===

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2019-06-24T08:20:35Z INFO:tensorflow:Graph was finalized. 2019-06-24 13:50:36.457345: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 2019-06-24 13:50:36.457398: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-06-24 13:50:36.457419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 2019-06-24 13:50:36.457425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N 2019-06-24 13:50:36.457539: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9855 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)

INFO:tensorflow:Restoring parameters from /tmp/tmpcla2n3gy/model.ckpt-40

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Evaluation [10/100]

INFO:tensorflow:Evaluation [20/100]

INFO:tensorflow:Evaluation [30/100]

INFO:tensorflow:Evaluation [40/100]

INFO:tensorflow:Evaluation [50/100]

INFO:tensorflow:Evaluation [60/100]

INFO:tensorflow:Evaluation [70/100]

INFO:tensorflow:Evaluation [80/100]

INFO:tensorflow:Evaluation [90/100]

INFO:tensorflow:Evaluation [100/100]

INFO:tensorflow:Finished evaluation at 2019-06-24-08:20:44

INFO:tensorflow:Saving dict for global step 40: _focal = 0.0016880237, _smooth_l1 = 0.0, dummyAccuracy = 100.0, global_step = 40, loss = 0.0016880237

This line :

==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@===
Tensor("IteratorGetNext:1", shape=(?, 120087, 5), dtype=float32, device=/device:CPU:0)
Tensor("regression/concat:0", shape=(?, ?, 4), dtype=float32)
==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@===
==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@===
Tensor("IteratorGetNext:2", shape=(?, 120087, 2), dtype=float32, device=/device:CPU:0)
Tensor("classification/concat:0", shape=(?, ?, 1), dtype=float32)
==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@===

shows that Accuracy2() is invoked two times first for regression then for classification. but I want that it gets invoked once with regression and classification fed into it together

$\endgroup$
2
$\begingroup$

I have found suggestion to implement metrics through callback.

https://github.com/keras-team/keras/issues/4506

Maybe it helps you and those with similar problem to figure out the needed solution

$\endgroup$
  • $\begingroup$ Cool ! looks helpful $\endgroup$ – Pratik Kumar Sep 4 '19 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.