Ability to locate original data after resuming a training

[kmeirlaen]

I want to continue training based on a .resume file. However, the source data (images / camera locations) are no longer in the location from the time the .resume file was created. In such a case, currently, there is no feedback to the user that the loading of the checkpoint and related data has failed or what is the cause. As a user, I would expect the software to

• warn the user that LFS could not find the orignal data and training cannot be resumed (minimum requirement)
• warn the that LFS could not find the original data and ask the user to re-locate the data by opening a file browser and allow the user to select the directory that contains the data. After user has selected the folder, validate the data and confirm if training can be resumed.

Same process for other paths (output directory) -> what to do when output directory no longer exists? Or should we just not always ask for the output directory when opening a .resume file? We cant assume the user still has the same directory structure (renaming, moving files, ...)

Alternatives:

• warn the that LFS could not find the original data, and look in related paths (by default, checkpoints are created in <base_data>/output/checkoints so LFS could verify if the expected data exists in the parent directory structure. If it found, it will inform the user that the source directory was identified
  or
• dont store full paths, but relative paths in the .resume file ? Dont like it?

Current console message: