How to access the logged results such as losses

In a simple training setup, I would like to directly access the lists/dicts of losses and other metrics logged during training and validation so that I can make some custom plots. Where can I find these stored?

As far as I understand you need to use such callback:

from pytorch_lightning import Callback
class MetricsCallback(Callback):
    """PyTorch Lightning metric callback."""

    def __init__(self):
        super().__init__()
        self.metrics = []

    def on_validation_end(self, trainer, pl_module):
        self.metrics.append(trainer.callback_metrics)

@Erlemar

Unfortunatelly this does not work:

trainer.callback_metrics

Contains only whatever was passed to EvalResult as a checkpoint_on or early_stop_on parameter.

Interesting - this code worked for me when I used it with old API (dictionaries instead of EvalResult). I hope there will be a solution.

Yep, I just tried the old dict API, and it works. EvalResults do not.

1 Like

can you try: trainer.logged_metrics or trainer.logger_connector.logged_metrics?

2 Likes

Since I am returning dictionaries for now instead of Result objects, the solution suggested by @Erlemar works.

But it does look like we need clarity around a solution for collecting metrics from Train or EvalResults going forward, instead of dicts. If I get to try it out, I will update this thread.

2 Likes

@goku

Thanks, trainer.logged_metrics works in both on_epoch_end and on_validation_end

It is still unclear to me what metrics can be found under trainer.logged_metrics in each callback, specifically are those reduced (by the on_epoch=True param in result.log) metrics or unreduced metrics.

It would be awesome to have this clearly described in the docs.

3 Likes