I’m working on a representation learning problem and the only way I really have of knowing if my model generalised is to forward the checkpoint to a few supervised learning problems and seeing the latents performance on each supervised task. Is it possible to asynchrnously submit a job after each epoch and then log the final result when the models are finished?
I appreciate this questions might be beyond the scope of pytorch lightning itself but it would be good to know if there’s a way to make the model “listen” for an external task to finish and log the outcome. Is it also possible to send a “hook” to the logger to an external process?