Multi-gpu training, weird tensorboard loss image

1

I trained the model with two gpus and use self.log('train_loss', loss) to log my loss in training_step.

There’re two losses every step which made it hard to read. Any solution to solve the problem?

I install lightning by pip install git+https://github.com/PytorchLightning/pytorch-lightning.git@master --upgrade yesterday. I think it’s 0.10.0rc1.

image
image1543×509 29.4 KB

2

Hello, my apology for the late reply. We are slowly converging to deprecate this forum in favor of the GH build-in version… Could we kindly ask you to recreate your question there - Lightning Discussions