Dynamic input

• Difficulty level: intermediate
• Time need to lean: 20 minutes or less
• Key points:
  • dynamic input targets are resolved only when the step is executed

dynamic input files

In order to determine the best execution strategy, SoS evaluates all expressions for all steps before the execution of a workflow to figure out input and output of steps. This works most of the time but sometimes the input of a step can only be determined at runtime. For example, if you would like your workflow to automatically scan an input directory and process all fasta files under it, or if a previous step produces files that cannot be determined beforehand, you might want to specify input files as follows,

input: 'input/*.fasta'

The problem is that no file or a wrong set files might exist during the planing stage so SoS might skip this step or start the step with a wrong set of files. To address this problem, you can declare the input files as dynamic by passing a dynamic object

input: dynamic('input/*.fasta')

This tells SoS that the input of this step can only be determined at runtime and will execute the step only after all its previous steps have been completed.

For example, step 10 of the following workflow generates some random output, and step 20 only knows it when this step is executed. If dynamic is not used, SoS will try to execute step 20 and 10 at the same time, and step 20 will not get the output of step 10.