Hello everyone,
I have a bit of unique situation and I am not sure if it is possible to implement within the Lightning framework.
Validation/testing in my model requires me to train another model to compute the validation/testing scores. To be more precise, my pipeline should work as follow: Step 1) train a model on the validation/testing dataset, Step 2) reuse the validation/testing dataset with the trained model to compute final validation/testing scores.
My initial thought is to do the training I need in “on_validation_epoch_start” and then do the score computation in “validation_step”. But for this I will need access to val_dataloader in “on_validation_epoch_start”. Is there a way to access the dataloader? Or is there a way to run the entire validation loop twice after each training loop with a flag?
Any suggestions would be very helpful?