Parallel artifact builds

vivarium.cluster_tools.core.jobmon.artifact.build_artifacts_in_parallel(*, workflow_name, build_commands, native_specification, worker_logging_root, env_prefix, resume=False, max_attempts=2, max_concurrently_running=None)[source]

Build location artifacts in parallel as a single Jobmon workflow.

Each (name, command) in build_commands - one per location - becomes an independent Jobmon task with no upstream dependencies, so the locations build concurrently (up to max_concurrently_running).

Return type:

tuple[str, str | None]

Parameters:

  • workflow_name (str) – Name (and workflow_args) identifying the Jobmon workflow. Use a fresh, unique name to start a new build; reuse a prior run’s name together with resume=True to resume that workflow.

  • build_commands (dict[str, str]) – Mapping of task name (e.g. "<location>_artifact") to the shell command that builds that location’s artifact.

  • native_specification (NativeSpecification) – SLURM resource request shared by every location’s build task.

  • worker_logging_root (Path) – Directory Jobmon writes per-task worker logs under.

  • env_prefix (str) – Absolute prefix of the conda env whose bin is prepended to PATH so each build command’s interpreter resolves without conda.

  • resume (bool) – If True, resume the workflow with the same workflow_name instead of starting fresh: Jobmon skips the location builds that already completed and reruns only the unfinished ones. Requires workflow_name to match the original run.

  • max_attempts (int) – Times Jobmon retries each location’s build before giving up.

  • max_concurrently_running (int | None) – Cap on locations building at once. None lets Jobmon decide.

Returns:

A (workflow_status, monitoring_url) tuple from client.bind_and_run_workflow.

Raises:

  • ValueError – If build_commands is empty.

  • RuntimeError – If the workflow finishes in any state other than complete.