Sacred Workflows

[Qwlouse]

In an attempt to structure our discussion I suggest to use this issue to collect a wishlist of how we would like to use Sacred from a birds-eye perspective.
I suggest that we edit this issue to reflect the evolving consensus that (hopefully) emerges from the discussion below.
To get things started I can think of 3 basic workflows, that I would love for sacred to support.
Maybe this is also a good place to think about how to integrate stages and superexperiments.

Interactive (Jupyter Notebook)

Manually control the stages of the experiment / run in an interactive environment. Most suitable for exploration and low complexity experiments. Something like:

# -----------------------------------------------------------
# initialization
ex = Experiment('my_jupyter_experiment')
ex.add_observer(FilestorageObserver('tmp'))
# -----------------------------------------------------------
# Config and Functions
cfg = Configuration()
cfg.learn_rate = 0.01
cfg.hidden_sizes = [100, 100]
cfg.batch_size = 32

@ex.capture
def get_dataset(batch_size):
    ....
# -----------------------------------------------------------
# run experiment
ex.start()   # finalizes config, starts observers
data = get_dataset()  # call functions 
for i in range(1000):
    # do something 
    ex.log_metric('loss', loss)  # log metrics, artifacts, etc.

ex.stop(result=final_loss)
# -----------------------------------------------------------

Scripting

Using a main script that contains most of the experiment and is run from the commandline.
This is the current main workflow, most suitable for low to medium complexity experiments.

ex = Experiment('my_experiment_script')

@ex.config
def config(cfg):
    cfg.learn_rate = 0.01
    ...

@ex.capture
def get_dataset(batch_size):
    ....

@ex.automain  # define a main function which automatically starts and stops the experiment
def main():
    ....   # do stuff, log metrics, etc.
    return final_loss

Object Oriented

This is a long-standing feature request #193. Define an experiment as a class to improve modularity (and support frameworks like ray.tune). Should cater to medium to high complexity experiments.
Very tentative API sketch:

class MyExperiment(Experiment):
    def __init__(self, config=None):   # context-config to deal with updates and nesting
         super().__init__(config)
         self.learn_rate = 0.001   # using self to store config improves IDE support
         ...

    def get_dataset(self):  # no capturing because self gives access to config anyways
        return ...

    @main   # mark main function / commands 
    def main_function(self):
         ...   # do stuff
         return final_loss

ex = MyExperiment(config=get_commandline_updates())
ex.run()

[davebulaval]

*edited (see after - )
First, what is the main objectif of Sacred ? (and the one we want to keep) - see issue about that
Second, what are the force of Sacred ? (and the one we want to keep or add ?!?) - Bad traduction -> the strong points, the must use (I think it's the config workflow and the automatic logging, but maybe there others points I miss)
Third, was is the intended level of the end-user (lambda or better) ? - Lambda user been user who don't know well Machine Learning (or else use of sacred) meaning some low level programmer user. And better been more advance programmer user knowing how to open the code and make it work their way. This will be scope the objective and level to reach for the framework (I think)

About OO approach

About Jupyter

Is it really necessary in a interactive environment to have a config file ? - @Qwlouse The expected workflow would be to use a Jupyter as an exploratory environment and afterwork transfert in a script environment ?

[thequilo]

For me, it would be important that everything is usable without sacred for various reasons (as already mentioned in #610 (comment)). So, captured functions should be callable without the need of a sacred experiment, configuration objects should be resolvable without an experiment or arg parser and an object-oriented experiment should be usable without the sacred environment by just passing the config values as arguments to init.

About OO approach

I have some questions about how you imagine the object-oriented experiments:

• Is the experiment class that is subclassed here the same as the sacred experiment that takes a name (and currently ingredients and others) as parameters for __init__?
• Is the __init__ supposed to work in the same way as a config scope? If so, this requires some black magic. To make it intuitive, it could be better to let __init__ have "normal" arguments.
• Does this support multiple inheritance? E.g., inherit from sacred.Experiment and some custom, say,Trainer?
• In my opinion, the overhead/impact of adding sacred to an already existing script should be as minimal as possible

To make it clearer:

class MyExperiment(Experiment, Trainer):   # Is this possible without strange MRO things happening?
    def __init__(self, learn_rate=0.001, ...):   # Pass "normal" args so that the impact of sacred is minimal
         super().__init__()    # Where does the experiment get its name from?
         self.learn_rate = learn_rate   # This is closer to the "sacred-free" use case
         ...

    @main
    def main_function(self, some_additional_arg):  # This should support additional args
         ...   # do stuff
         return final_loss

ex = MyExperiment(**get_commandline_updates())  # This now becomes difficult. Is it even possible?

# What about config updates from outside? This makes the use of `__init__` for config creation even
# more difficult
ex.config.add_config(learn_rate=0.1)

# Wouldn't this usually get the command line updates?
ex.run()

And it should be possible to do

ex = MyExperiment(learn_rate=1234)
print(ex.main_function(some_additional_arg=2345))

About Jupyter

Additionally, it should be no problem to import an experiment (class) or a config object from an existing script (i.e., it should not be required to pass any additional interactive=True flag to anything)

[flukeskywalker]

@thequilo It is unclear to me what you mean by everything being usable without Sacred (after reading your linked comment). I think we all agree that one should be able to use, e.g., gin-config + sacred for logging all information about the experiment. But it seems you mean something more(?)

Do you specifically mean without defining a Sacred Experiment (while still using Sacred otherwise, e.g. just for configuration resolution) instead of without sacred?

[thequilo]

This is now not a so big problem anymore with the config and experiment decoupled from each other. It is actually quite difficult to describe and depends on what exactly you do. Over the past I multiple times ran into the problem that I had to construct a dummy experiment to be able to use some part of the experiment in the notebook or in tests.

For captured function, I mean it should be possible to call them without constructing an Experiment. This simplifies testing and "quick tests" in a notebook a lot (this is currently possible if no captured functions are used from within captured functions). Same for config object, should be possible to construct without an experiment and command-line interface and also based on other frameworks (gin-config). This is currently not possible but will be possible after the rework. For the experiment class it should be possible to instantiate it without constructing a config (like: pass some arguments to its init, and this does not evaluate config scopes because they are decoupled) and without command-line.

If it is still too confusing what I mean, we can just ignore it. I feel like the rework discussion is going in the right direction with this and I'll comment if something is going to become "too involving" or unusable "without sacred".

[thequilo]

Superexperiments and Stages

Since @Qwlouse said that this could be the place for discussions about Superexperiments and Stages, here a small proposal of what could be possible with the reworked config. This does not require anything new (in addition to the things that already emerged from the discussion), but it would be convenient to have a way of registering a run/experiment as a sub-run (or sub-experiment) or another one. This could even be nested.

So, as a very rough idea (ignore any naming): A "Stage" is an experiment that is used by a "Superexperiment". A run object can be registered to another run object to be a sub-run of that run, this works nested as well.

data_stage = Experiment('data_preparation')
train_stage = Experiment('train')
eval_stage = Experiment('eval')

# Now define your experiments with captured functions, config, possibly in different files...
@data_stage.main
def data_main():
    if check_if_already_completed():
        # Not sure how to do this yet, maybe check for specific files or the state of some 
        # experiment ID in a db
        return
    else:
        # Run data preparation

@eval_stage.main
def eval_main():
    if check_if_already_completed():
        # Could return the result of the completed run to make restarting of failed or 
        # incomplete parameter sweeps possible
        return load_result_of_completed_eval_run()
    else:
        # Run evaluation
        return result

ex = Experiment('superexperiment')

@ex.config
def config(cfg):
    stft_size = 512   # An example config value that gets shared among all stages

    # Add the config of the stages like for ingredients
    # This makes sense in this case, but it some cases we might want to share the whole
    # config among stages. See below
    cfg.data = data_stage.config(stft_size=stft_size)
    cfg.train = train_stage.config(stft_size=stft_size)
    cfg.eval = eval_stage.config(stft_size=stft_size)

# To share the whole config among all stages. Then, it does not make sense to add the config
# as for ingredients as above
data_stage.config.add_config(ex.config)
train_stage.config.add_config(ex.config)
eval_stage.config.add_config(ex.config)

@ex.automain
def main(_run, _config):
    # We could add a "super_run" arg to the run method so that the resulting run object 
    # (if we keep something like that) is registered as a sub-run or sub-experiment
    data.run(super_run=_run)

    # In some cases, it might make sense to allow passing the config and an additional 
    # ID to run (e.g., paramter sweeps)
    results = []
    for learning_rate in (0.1, 0.01, 0.001):
        train_run = train.create_run(
                                super_run=_run, 
                                config_updates=dict(learning_rate=learning_rate),
                                stage_id=f'run_learning_rate={learning_rate}')
        train_run.run()
        
        # Pass the constructed config of the train run to the eval run so that eval knows where
        # the model files are stored
        eval_run = eval.create_run(
                                super_run=_run, 
                                config_updates=train_run.config, 
                                stage_id=f'eval_learning_rate={learning_rate}')
        result = eval_run.run()
        results.append((result, eval_run))

    # And now we can find the optimal configuration
    best = min(results, key=lambda x: x[0])
    return best

What do you think? Does this make any sense?

[flukeskywalker]

Let me clarify here that the main reason that Sacred Experiments are not compatible with ray.tune's class based API (for example) is not exactly the lack of a OO API, but a bit deeper.

Currently Sacred can be used with the functional API of ray.tune as follows (a bit simpler than @rueberger's #193 (comment)):

def train_example(config, reporter):
    from my_script import ex
    ex.observers.append(MongoObserver.create(...)
    result = ex.run(config_updates=config)
    reporter(...)

The (class-based) Trainable API of ray.tune is more powerful because it is designed with the understanding that experiment runs can be structured like calling a function step() repeatedly (potentially with different config updates), which provides a good point to save or restore an experiment if needed, and return control to the calling scope.

I haven't tested this yet (Update: tested, works), but I realized that actually it may actually be possible to hack a Trainable class using Sacred:

import ray
from ray import tune
from ray.tune import Trainable

class Example(Trainable):
    def _setup(self, config):
        from my_script import ex
        self.ex = ex
        self.current_config = config

    def _train(self):
        run = self.ex.run('step', config_updates=self.current_config)
        return dict(value=run.result)

    def _save(self, tmp_checkpoint_dir):
        config_updates = {'tmp_checkpoint_dir': tmp_checkpoint_dir}
        run = self.ex.run('save', config_updates=config_updates)
        return run.result

    def _restore(self, checkpoint_path):
        config_updates = {'checkpoint_path': checkpoint_path}
        run = self.ex.run('restore', config_updates=config_updates)

    def reset_config(self, new_config):
        self.current_config = new_config

Here step(), save(), restore() are the Commands defined for the Experiment. The main "hack" here is hidden but in my_script, the user would need to attach any persistent objects (such as Network and Optimizer objects) to the Experiment object itself. The big loss is that it can't solve the problem of observers; they will create a new run for each call of each command :(
One could use tune's own system for tracking of metrics as a stop-gap.

In summary, I think an OO design of Experiment would need to:

1. formalize the hack above (some objects will need to be persistent across commands)
2. reinterpret the concept of calling a command (not always a new run)
3. add support for saving/restoring Experiments (including observers)
4. record events such as changes in config, changes in the host & environment etc.

[thequilo]

For "reinterpret the concept of calling a command": It would be useful for any kind of parallel experiment, e.g., MPI

[Qwlouse]

@davebulaval

I do not understand what you mean by "force of Sacred" or by the expected level of the end user (what is lambda in this context?). But it is a good point, that we should write a goal / vision statement for sacred. I'll draft something and post it for discussion.

One drawback I see with the actual propose API class is that code won't seem to be reusable that much with the main_function method.

The main function is only the default command, and you are free to define others.

Is it really necessary in a interactive environment to have a config file ?

Yes. An important reason to use sacred in an interactive environment would be to draft an experiment that can later be converted into a script. Also I might still want to use observers that log the configuration etc.

@thequilo

For me, it would be important that everything is usable without sacred

I understand and completely agree: This is a very important point. Ideally each component of sacred (the configuration, the command/capture functions, observers, the commandline interface, ...) should be usable in isolation. That way you can use and test them individually, nothing gets "locked-in" to an experiment, and it provides a lot of flexibility for customization. All of these points are very important in my opinion, and we could do a much better job of enabling them.

I have some questions about how you imagine the object-oriented experiments [...]

These are very good questions, by which I mean: I do not have good answers :-)

• If possible I would like to keep the three usecases (interactive, script, OO) as similar and interchangeable as possible. But I am not sure how feasible that will be.
• There should be some way of getting the same behavior as @ex.config in the OO case, but it doesn't have to be the __init__ method. Reasons for using the __init__ include: Better IDE support, and the intuitive simplicity of just defining the configuration as member fields of the experiment.
• Multiple-Inheritance: That is a good point we should keep in mind. No idea yet as to the consequences.

RayTune

@flukeskywalker Wouldn't it be enough then to do someting like this?:

from my_script import ex, step, save, restore

class Example(Trainable):
    def _setup(self, config):
        from my_script import ex
        self.ex = ex
        self.run = ex._create_run(config_updates=config)

    def _train(self):
        return step()

    def _save(self):
        save()

    def _restore(self):
        restore()

    def reset_config(self, new_config):
        self.run = ex._create_run(config_updates=new_config)

This creates a new run upon initialization and everytime reset_config is called. The captured functions step, save, and restore use the configuration from the current run. This is obviously not thread-safe (this is a general problem with sacred), but it would keep the observers alive.

[thequilo]

I think one problem with raytune is that for some distributed configurations it can run one step one one machine, copy the saved state to another machine and resume there for the next step (I hope I understood the docs correctly). Then, your approach above does not work and it would be necessary to implement some sort of resume mechanism for the observers (And this kind of brings back the discussion about the Observers/Retriever #483, because resuming is essentially the same as loading data from a storage for analysis).

[flukeskywalker]

Note: I've corrected my code snippet and tested that it actually works.

@Qwlouse Unfortunately that doesn't work. The global imports make ray attempt to serialize those objects, which ends in disaster. This is one of the reasons that it took a while for people to trust that ray & Sacred can work together (CC @engintoklu). Making each ray actor do the imports locally is the only way I've figured out to avoid this issue.
Even if it worked, it is debatable that a change to config should reset observers instead of just continuing.

In general (and as @thequilo points out above), medium to large experiments have a general requirement of checkpointing/restoring to handle failed experiments on clusters etc. So the 4 requirements above seem unavoidable.

I am not sure whether it is just the observers that need to be modified or the Experiment/Run behavior as well. Would something like ex.run(..., reset_observers=False) together with ex.save(...) and ex.restore(...) be sufficient?

[Qwlouse]

@thequilo and @flukeskywalker: Regarding Stages and RayTune:
This problem is ties into many other things that should be improved about Sacred. It ties in with an earlier discussion about continuing an experiment #411 and about supporting population based training (see eg this comment by @rueberger )
I think @csadorf summarized it nicely by pointing out that the main problems are:

1. There is no concept of data persistence in sacred (see also retrievers Observers counterpart (retrievers?) #483)
2. There is no concept of workflow lineage or dependency in sacred.

I would add to that:

3. Thread safety to allow parallel execution of multiple runs from the same program
4. Full pickleability to support serialization and distributing across machines
5. An overhaul of the observer data format to account for stages, resuming, and changing parameters during a run.

These are important issues, and I emphatically share the wish to support these. However, I am afraid, that tackling them all at once is too large a project. It will require re-engineering several core parts of Sacred apart from the configuration system. I'm happy to discuss them, and gladly welcome any input on these issues. But sadly, I doubt that we have the capacity to tackle them at the moment.

[thequilo]

@Qwlouse Agreed. We should for now focus on the config process. A discussion about data persistence and observers would by itself become as big as the discussion about the config that we already have. But it might be good to have some place where general ideas can be collected (Another issue) for future work.

While reworking the config we should make sure that point 4 is fulfilled (full pickleability of the config object(s)), and maybe also point 3 (thread safety) for potential captured functions and the config creation process.