row_log_interval from Trainer flags do?
I noticed logging in tensorboard is done at
row_log_interval (default 50) while in wandb it doesn’t.
Why not for wandb?
I’m looking at the trainer code and it is applied to whatever logger is used:
should_log_metrics = (batch_idx + 1) % self.row_log_interval == 0 or self.should_stop if should_log_metrics or self.fast_dev_run: # logs user requested information to logger metrics = batch_output.batch_log_metrics grad_norm_dic = batch_output.grad_norm_dic if len(metrics) > 0 or len(grad_norm_dic) > 0: self.log_metrics(metrics, grad_norm_dic)
row_log_interval is for how many steps should be skipped before adding a new row to the summary.
log_save_interval is the interval at which this summary is actually written to disk. this is the slowest part, and determines how often you can “refresh” your tensorboard.
log_save_interval may not apply to all loggers if they do their own thing, like sending the data to the cloud.
By summary you mean all the logs from previous steps are stored and at log_save_interval, it saves them all to the disk the clear up summary state of TensorboardLogger? So with this logger if I set
log_save_interval=1000 for total batches of 100, it won’t create any logs, right?
Yes, row_log_interval applies to all logger.
I changed x-axis to
step in the wandb plot so it showed from 0, 1, 2, …
So I miss understood it