I am trying to calculate a metric on the entire validation set. The metric can not be computed on batches and can not be approximated from per_batch results. Ideally I would like to write checkpoints depending on this epoch level metric.
So far, I have found two options:
A) Add a Callback that runs a fn with a pass over the validation set to calculate the metric. I know that it is possible to log the metric to
loggers from here (e.g. Tensorboard). But how would I add it to the Checkpoints-Callback? Is it possible to write to the Evalresult obj from other callbacks?
on_validation_epoch_end(). The same question as in A) applies.
What is the best way to implement this?
Many thanks for your help!