Multi-Modal models

1

How would this be done?

  • 2 or more datasets will need to be combined, at least for the training
  • multiple losses
  • 2 or more models need to be combined
    • it would be nice to be able to plug-and-play, swap out different models for each mode
    • lets generalise that to any kind of multi-model, where we can combine models in all sorts of ways.
  • the training regimes for each model maybe different than the others

Thanks

2

For the datasets, the Lightning DataModule might be a good choice. You can parameterize it with hyperparameters for your dataset assembly. Maybe treat it like a factory.

For the model, maybe pass it into the LightningModule as arguments. You can then call this LightningModule a system or a task. You can define a metric and run several trainings with different models and datamodules to test which model performs best.