Rocoto won't run second Metatask after first completes

[uwagura]

I currently have a Rocoto workflow that runs two sets of Metatasks in serial using versino 1.3.7 of Rocoto. My goal is to have the second set of metatasks begin once all of the first metatasks complete, so I set a metataskdep for the first task in my second set of metatasks to accomplish this. However, once the first set of tasks completes, running rocotorun a second time does not launch the second set of tasks. Is there something wrong with my workflow configuration that is causing this error?

The second set of metatasks does not depend on a particular cycle, it just needs to run after all cycles in the first metask are complete, so I set a "dummy" cycle def to try and accomplish this.

<workflow realtime="False" scheduler="slurm" cyclethrottle="326" taskthrottle="326"> <!-- cyclethrottle needs to be greater than ens*years-->
  <cycledef group="metatask_1"> 0 0 1 1 1960-1962 * </cycledef>
  <cycledef group="dummy_cylce"> 0 0 1 1 3000 * </cycledef> <!-- Dummy date to get ensemble tasks to run after year + ensemble tasks -->
  <log>
    <cyclestr>/work/&USER;/&DOMAIN;/job_output_files/rocoto_logs/Decadal_Workflow_&DOMAIN;_@Y@m@d@H.log</cyclestr>
  </log>

  <!-- first metatask, which needs to run for several ensemble members and several years -->
  <metatask name="a1_atm_and_oce">
    <var name="ens"> &ENSEMBLES; </var>
    <task name="generate_oce_atms_#ens#" cycledefs="metatask_1">
      <nodes>1:ppn=1</nodes>
      <partition>batch</partition>
      <walltime>43200</walltime>
      <command>
        <cyclestr>generate_historical_atm_and_oce.sh -e #ens# -y @Y -w &WORK; -g &GAEA;</cyclestr>
      </command>
      <jobname>generate_historical_atm_and_oce.sh</jobname>
      <join>./job_output_files/generate_historical_atm_oce_%j.job</join>
     </task
 </metatask>

<!-- metatask to run after all tasks in previous step complete -->
  <metatask name="concat_and_move_a1">
    <var name="ens"> &ENSEMBLES; </var>

    <task name="concat_and_move_obcs_#ens#" cycledefs="dummy_cycle" >
      <nodes>1:ppn=1</nodes>
      <partition>batch</partition>
      <walltime>43200</walltime>
      <command>
        <cyclestr> 10_write_spear_a1_oce_concat_mv.sh -e #ens# -w &WORK; </cyclestr>
      </command>
      <jobname>10_write_spear_a1_oce_concat_mv</jobname>
      <join>./job_output_files/10_write_spear_a1_oce_concat_mv_%j.sh</join>
      <dependency>
        <metataskdep metatask="a1_atm_and_oce"/>
      </dependency>
    </task>

    <task name="stage_oce_atms_#ens#" cycledefs="dummy_cycle" >
      <nodes>1:ppn=1</nodes>
      <partition>batch</partition>
      <walltime>43200</walltime>
      <command>
        <cyclestr>12_cp_gfdl_gaea_spear_a1_oce.sh -e #ens# -w &WORK; -g &GAEA; </cyclestr>
      </command>
      <jobname>12_cp_gfdl_gaea_spear_oce</jobname>
      <join>./job_output_files/12_cp_gfdl_gaea_spear_a1_oce_%j.job</
      <dependency>
        <taskdep task="concat_and_move_obcs_#ens#"/>
      </dependency>
    </task>

  </metatask>

[christinaholtNOAA]

Hi @uwagura, taking a quick look at your example XML, it looks like the problem with the second metatask not running is the cycledefs entry in the second set of tasks. You will need all the tasks in the workflow to be valid on the same set of cycles to have them run in the order you describe.

Try a change like this:

    <task name="concat_and_move_obcs_#ens#" cycledefs="metatask_1" >
...
    <task name="stage_oce_atms_#ens#" cycledefs="metatask_1" >

[uwagura]

Thanks for getting back to me @christinaholtNOAA . While setting the second set of tasks to have the same cycledef as the first set does work, it does not produce the workflow that I am looking for.

I'm trying to get the second metatask, which only iterates over ensemble members, to run after all of the tasks in the first metatask complete which iterates over both ensemble members and years within the cycledef. If I set the second metatask to run over the same cycle defs as the first, it will iterate over both years and ensembles, which I do not want.

Is there a way to achieve this structure using rocoto?

[christinaholtNOAA]

I think I understand. Let me check...you'd like to run a whole data set over multiple years, and when that's done, do some sort of data gathering or post-processing on the entire dataset.

If that's the case, then unfortunately, I don't think this is the best way to set up your dependencies. Rocoto operates in a very cycle-centric way by default. What I mean is that the metatask dependency means that once all the tasks in a metatask for a cycle are complete, you can kick off a given task in that cycle. For example, if you gathered all the ensemble members at a given cycle and wanted to postprocess that single ensemble, you'd use a metatask dependency for that one ensemble.

There are a couple of options to achieve what I think you're after:

1. Use a nested set of metatasks for running generate_historical_atm_and_oce.sh where the outer can loop over ensemble members, and the inner can loop over years. You'd need to change your argument to the script from @Y to #year# where your var for that metatask is a list of years. In this case, you'd want to use your dummy cycledef for all tasks.

Your second metatask could then use the inner metatask as a dependency. Here's an untested example:

 <metatask name="a1_atm_and_oce">
    <var name="ens"> &ENSEMBLES; </var>
    <metatask name="a1_atm_and_oce_all_times">
     <var name="year"> 1960 1961 1962</var>
        <task name="generate_oce_atms_#ens#" cycledefs="dummy_cycle">
    ...

  <metatask name="concat_and_move_a1">
    <var name="ens"> &ENSEMBLES; </var>

    <task name="concat_and_move_obcs_#ens#" cycledefs="dummy_cycle" >
...
      <dependency>
        <metataskdep metatask="a1_atm_and_oce_all_times"/>
      </dependency>
    </task>

2. You can split the two original metatasks into separate workflows and only start the second when all the cycles from the first are complete.

3. You can use data or shell dependencies instead of metatask dependencies to check that all the output of the first metatask is available before you start the tasks.

I might still be on the wrong track with my understanding of your ideal workflow. Let me know.

[uwagura]

@christinaholtNOAA , Thanks for the clarification. I'll probably pursue one of the other options then for the specific task I'm trying to accomplish.

[christopherwharrop-noaa]

Thank you @christinaholtNOAA for helping answer this and for providing great suggestions. @uwagura, unfortunately, you have stumbled upon a limitation in Rocoto's ability to express workflows. It doesn't handle tasks that run outside of the scope of a cycle very well.