If I use the code from the documentation (https://pytorch-lightning.readthedocs.io/en/stable/logging.html#logging):
tb_logger = pl_loggers.TensorBoardLogger('logs/') csv_logger = pl_loggers.CSVLogger(save_dir='logs/') trainer = Trainer(logger=[tb_logger, csv_logger])
and enable ddp in the Trainer
distributed_backend='ddp', I get a folder structure like this:
logs --default ----version_1 ----version_2
with an version_x folder for each ddp process. Thats quite confusing and does not make any sense (to me)
Is there a way to set a path for the entire trainer once and have the loggers and checkpoints to that path or subdirs of that path?
I tried to have a fixed version with
version=str(datetime.datetime.now()) but this does not work as the other DDP processes start at slightly different times.