Hi,
I have an episodic few-shot dataset, and I want to iterate over each episode and train a new pre-trained T5 model in each iteration, the best model is chosen by the min val_loss among all the models.
- If I load the pre-trained T5 and the checkpoint outside the loop and then create a new trainer in the loop does the checkpoint continue the previous model or save the checkpoint of the new model?
- should I reset the trainer after each iteration → trainer.fit()?
checkpoint_callback = pl.callbacks.ModelCheckpoint(dirpath=f'{opt.output_dir}/train', monitor="avg_val_loss", mode="min")
early_stopping_callback = EarlyStopping(monitor="avg_val_loss")
model = T5FineTuner(opt=opt)
# iterate over the training episodes
for x in range(len(train_data)):
# save_top_k=1,
train_params = dict(
accumulate_grad_batches=opt.gradient_accumulation_steps,
gpus=opt.n_gpu,
max_epochs=opt.num_train_epochs,
precision=16 if opt.fp16 else 32,
amp_level=opt.opt_level,
gradient_clip_val=opt.max_grad_norm,
log_every_n_steps=10,
callbacks=[LoggingCallback(), early_stopping_callback, checkpoint_callback],
)
train_dataloader = model.train_dataloader(train_data[x])
val_dataloader = model.val_dataloader(train_data[x])
trainer = pl.Trainer(**train_params, default_root_dir=f'{opt.output_dir}/train')
trainer.fit(model, train_dataloaders=train_dataloader, val_dataloaders=val_dataloader)
best_model_path = checkpoint_callback.best_model_path
best_score = checkpoint_callback.best_model_score