Validation Error: The validation bar turns red and no callbacks are called, no checkpoints are saved

I currently code in kaggle kernel for a competition where I found this weird behavior.
My validation progress bar turned red and no checkpoints were saved but training proceeded to next epoch while validation was struck.

mind share the notebook or reproduce this behavior with

1 Like

The link to my notebook

The validation loop is not stuck, this behaviour is just caused by progress_bar_refresh_rate. Your validation loop has 263 steps and the progress bar gets updated only after every progress_bar_refresh_rate (20) steps. So, last it (progress bar) got updated is after 260 steps, because it is divisible by 20. The remaining 3 aren’t updated in that progress bar but the model has completed them.

1 Like

thank you for your great explanation. I understood what it is

I also tried it, working fine for me. The validation progress bar disappears as expected. Also, I used limit_train_batches=23 just to replicate your case and avoid long runs.