Workflow Config Interface

Python API for building workflow step tasks programmatically, as an alternative to authoring a YAML workflow file. Each function validates its kwargs and constructs the Jobmon tasks for that step type.

vivarium.cluster_tools.dagger.config.interface.get_step_resources(*, memory_gb, project, queue, runtime='01:00:00', cores=1, hardware=None, requires_archive_node=False)

Build the compute resources for a workflow step.

Pass the return value as the resources argument to any get_*_step_tasks function. project and queue are required.

Return type:

ResourceConfig

Parameters:

• memory_gb (int) – Memory in GB.

• project (str) – Cluster project to charge.

• queue (str) – Cluster queue to submit to.

• runtime (str) – Maximum runtime in hh:mm:ss format. Default is 01:00:00.

• cores (int) – Number of CPU cores to request. Default is 1.

• hardware (list[str] | None) – Optional list of hardware types to target (e.g. ["r650"]).

• requires_archive_node (bool) – Whether to enforce landing on an archive node. Default is False.

Returns:

The compute resource specification for a step.

vivarium.cluster_tools.dagger.config.interface.get_bash_step_tasks(*, name, resources, command, output_directory, tool, environment=None)

Build a bash workflow step and return its Jobmon tasks.

Return type:

list[Task]

Parameters:

• name (str) – Unique name for this step within the workflow.

• resources (ResourceConfig) – Compute resources for this step.

• command (str) – Shell command string to execute.

• output_directory (Path) – Directory for this step’s worker logs and step-level outputs.

• tool (jobmon.client.api.Tool) – Jobmon Tool used to register task templates and create tasks.

• environment (str | None) – Optional conda environment name to use for this step. If unset, falls back to the runner’s active CONDA_DEFAULT_ENV.

Returns:

The Jobmon tasks produced by the step.

vivarium.cluster_tools.dagger.config.interface.get_simulation_step_tasks(*, name, resources, output_directory, model_specification, branch_configuration, tool, environment=None, artifact_path=None, backup_freq=1800.0, sim_verbosity=0, is_resume=False)

Build a parallel-simulation workflow step and return its Jobmon tasks.

Produces one Jobmon task per (input_draw, random_seed, branch) combination defined by the branch configuration. Uses the same task runner infrastructure as psimulate run.

Return type:

list[Task]

Parameters:

• name (str) – Unique name for this step within the workflow.

• resources (ResourceConfig) – Compute resources for each individual simulation task (memory, runtime, cores).

• output_directory (Path) – Directory for this step’s outputs. The simulation step lays out model_name / timestamp subdirectories beneath this.

• model_specification (Path) – Path to the model specification YAML file. Both relative and absolute paths are accepted.

• branch_configuration (Path) – Path to the branch configuration YAML file. Both relative and absolute paths are accepted.

• tool (jobmon.client.api.Tool) – Jobmon Tool used to register task templates and create tasks.

• environment (str | None) – Optional conda environment name to use for this step. If unset, falls back to the runner’s active CONDA_DEFAULT_ENV.

• artifact_path (Path | None) – Optional path to a data artifact file. Both relative and absolute paths are accepted.

• backup_freq (float | None) – Backup frequency in seconds, or None to disable backups. Defaults to 30 minutes.

• sim_verbosity (int) – Vivarium simulation logging verbosity level. Default is 0.

• is_resume (bool) – Whether this is a resumed workflow build.

Returns:

The Jobmon tasks produced by the step.

vivarium.cluster_tools.dagger.config.interface.get_pytest_step_tasks(*, name, resources, output_directory, tool, environment=None, path=None, k=None, runslow=False)

Build a pytest-based workflow step and return its Jobmon tasks.

At least one of path or k must be provided. When the step’s resources.cores is greater than 1, the command is run with --numprocesses <cores> (pytest-xdist).

Return type:

list[Task]

Parameters:

• name (str) – Unique name for this step within the workflow.

• resources (ResourceConfig) – Compute resources for this step.

• output_directory (Path) – Directory for this step’s worker logs and step-level outputs.

• tool (jobmon.client.api.Tool) – Jobmon Tool used to register task templates and create tasks.

• environment (str | None) – Optional conda environment name to use for this step. If unset, falls back to the runner’s active CONDA_DEFAULT_ENV.

• path (str | list[str] | None) – Test path(s) — a single file/directory or a list of them — passed to pytest as positional arguments. Both relative and absolute paths are accepted.

• k (str | None) – Pytest -k expression used to filter tests by name.

• runslow (bool) – If True, pass --runslow to pytest. Default is False.

Returns:

The Jobmon tasks produced by the step.

vivarium.cluster_tools.dagger.config.interface.get_python_step_tasks(*, name, resources, output_directory, path, tool, environment=None, positional_args=None, keyword_args=None)

Build a Python-script workflow step and return its Jobmon tasks.

Constructs a python <path> [positional_args...] [--key value...] command. Positional arguments are appended in list order; keyword arguments are emitted sorted by key. Keyword values map to CLI flags as follows:

• True or None -> bare --key flag

• False -> omitted from the command

• any other scalar -> --key value

Parameters:

• name (str) – Unique name for this step within the workflow.

• resources (ResourceConfig) – Compute resources for this step.

• output_directory (Path) – Directory for this step’s worker logs and step-level outputs.

• path (str) – Path to the Python script (must end with .py). Both relative and absolute paths are accepted.

• tool (jobmon.client.api.Tool) – Jobmon Tool used to register task templates and create tasks.

• environment (str | None) – Optional conda environment name to use for this step. If unset, falls back to the runner’s active CONDA_DEFAULT_ENV.

• positional_args (list[Any] | None) – Optional list of scalar values appended in order as positional CLI arguments. Defaults to none (empty).

• keyword_args (dict[str, Any] | None) – Optional dict mapping identifier-style keys to scalar values, rendered as --key value flags (see flag rules above). Defaults to none (empty).

Returns:

The Jobmon tasks produced by the step.

Return type:

list[Task]

vivarium.cluster_tools.dagger.config.interface.get_notebook_step_tasks(*, name, resources, output_directory, path, output_path, tool, environment=None, parameters=None, cwd=None)

Build a notebook-based workflow step and return its Jobmon tasks.

Parameter values map to papermill flags as follows:

• str / int / float -> -p key value

• bool / None -> -y key {true,false,null} (YAML-typed)

Parameter keys must be valid Python identifiers because papermill injects them as variable assignments in a notebook cell.

Parameters:

• name (str) – Unique name for this step within the workflow.

• resources (ResourceConfig) – Compute resources for this step.

• output_directory (Path) – Directory for this step’s worker logs and step-level outputs.

• path (Path) – Path to the input notebook (must end with .ipynb). Both relative and absolute paths are accepted.

• output_path (Path) – Path where the executed notebook will be written (must end with .ipynb). Both relative and absolute paths are accepted.

• tool (jobmon.client.api.Tool) – Jobmon Tool used to register task templates and create tasks.

• environment (str | None) – Optional conda environment name to use for this step. If unset, falls back to the runner’s active CONDA_DEFAULT_ENV.

• parameters (dict[str, Any] | None) – Optional dict of scalar values injected as notebook parameters.

• cwd (Path | None) – Optional working directory for notebook execution. If not provided, defaults to the parent directory of path.

Returns:

The Jobmon tasks produced by the step.

Return type:

list[Task]