Dynamic ouput

• Difficulty level: intermediate
• Time need to lean: 10 minutes or less
• Key points:
  • dynamic output targets are resolved after the step is completed

dynamic output files

Similar to the cases with dynamic input files, the output of some steps could also not be determined beforehand.

For example, with the following script that generates .out files that cannot be determined during dry run,

In this case, you will need to define the output as dynamic using a dynamic function.

Dynamic output from substeps

It is a bit tricky to make dynamic output work with substeps because substeps are supposed to produce distinct outputs, and expressions like temp_dynamic/*.out could grab output from multiple substeps. It is therefore recommended that the pattern in dynamic output vary among substeps as follows:

Dynamic expansion of non-existing files

Whereas dynamic function is usually used with a filename pattern, it can also be used to fixed filenames, which would be "expanded" to an empty list if the file does not exist.

This fact can be used to handle cases when the output might not be generated for some reason. For example, in the following workflow there is a 50% chance that output from a substep will not be generated. Instead of raising an error, use of dynamic around the output generates an empty output.

Note that you can remove empty substeps by regrouping step_input in the subsequent processing steps