Outcome-oriented workflows

• Difficulty level: intemediate
• Time need to lean: 20 minutes or less
• Key points:
  • Instead of executing workflows, you can use option -t to specify targets to generate
  • Targets can be filenames or names of named output
  • Steps that provides targets can be process-oriented steps with static or named outputs, or outcome-oriented with pattern matching

Outcome-oriented workflows

Outcome-oriented workflows aim at generating particular outcomes. The essential features include

1. The workflow consists of steps that provides "outcome" for other steps
2. The workflow is triggered by the generation of particular outcome

Steps that provides targets through pattern matching

A typical makefile-style auxiliary steps with a section header with a provides option, which usually contains a pattern. The pattern serves three purposes

1. It triggers the step through pattern matching. For example, provides='{filename}-{idx}.txt' matches a-1.txt, filename-23.txt, but not filename.txt.
2. It defines variables specified in the pattern. For the {filename}-{idx}.txt example, a-1.txt would generate variables filename='a' and idx='1'.
3. It defines a default output to the step, which would be output: 'a-1.txt' if the step is matched to filename a-1.txt. Note that the default output could be overriden by an explicit statement.

For example, the following workflow consists of one auxiliary step that generates .csv file from .xlsx file. The workflow is triggered by option -t data/DEG.csv (target) so a command

xlsx2csv data/DEG.xlsx data/DEG.csv

is executed to generate it. The data/DEG part is determined by pattern matching which assigns filename='data/DEG'.

Steps with simple output

If the output of a step is "simple", in the sense that it can be determined by itself without referring to any definition in the global section or the input of the step, it automatically provides the output to other steps.

For example, the plot step added to the previous workflow does not have a provides statement. It has a simple output statement with figure.pdf. Then, when the workflow is triggered by -t figure.pdf, the plot step will be triggered, which, in this case, also triggers convert because of the missing data/DEG.csv.

Generating outputs with named output

If the output is not simple (e.g. involves global definitions or parameters), or it contains multiple targets, you can give it a name through keyword argument (see named output for details). You can then specify the name to option -t instead of the actual filename(s).

For example, by assigning output to a name figure, the following workflow could be triggered with -t figure.