Vivarium Cluster Tools Documentation

Vivarium cluster tools is a python package that makes running vivarium simulations at scale on a Slurm cluster easy.

• Installation
• Running simulations in parallel
  • Restarting a Simulation
  • Expanding a Simulation
• YAML Basics
  • Structure
  • Comments
  • Mappings
  • Lists
  • Composite Data
• The Branches File
  • Uncertainty
  • Configuration Variations
• Dagger: Multi-Step Workflows
  • Quickstart
  • Workflow YAML schema
  • Running and restarting
  • How dagger behaves
• Logging
  • Top-level logs
  • Cluster logs
  • Worker logs
  • Performance logs
• API Reference
  • vivarium.cluster_tools
  • vivarium.cluster_tools Utilities
  • get_cluster_name()
  • hash_output_path()
  • mkdir()
  • backoff_and_retry()
  • core
  • Shared CLI tools
  • with_verbose_and_pdb()
  • with_sim_verbosity()
  • resolve_sim_verbosity()
  • with_slack_channel()
  • with_slack_tag()
  • with_slack_mute()
  • validate_slack_options()
  • coerce_to_full_path()
  • pass_shared_options()
  • MinutesOrNone
  • load_run_config()
  • with_run_config()
  • with_workflow_config()
  • resolve_deprecated_positional()
  • Logging Utilities
  • configure_main_process_logging_to_terminal()
  • configure_main_process_logging_to_file()
  • Cluster Interaction
  • Cluster CLI options
  • with_queue_and_max_runtime()
  • Cluster Interface
  • NativeSpecification
  • get_workflow_timeout_seconds()
  • dagger
  • dagger CLI
  • dagger Runner
  • run_workflow()
  • restart_workflow()
  • Workflow Configuration
  • Workflow Builder
  • STEP_TYPE_API_FNS
  • build_workflow_from_config()
  • Workflow Configuration
  • REQUIRED_WORKFLOW_FIELDS
  • DEFAULT_BACKUP_FREQ_SECONDS
  • ResourceConfig
  • ParsedStep
  • WorkflowConfig
  • Workflow Config Interface
  • get_step_resources()
  • get_bash_step_tasks()
  • get_simulation_step_tasks()
  • get_pytest_step_tasks()
  • get_python_step_tasks()
  • get_notebook_step_tasks()
  • Workflow Step Parsing
  • resolve_step_type()
  • parse_bash_step_from_yaml()
  • parse_simulation_step_from_yaml()
  • parse_pytest_step_from_yaml()
  • parse_python_step_from_yaml()
  • parse_notebook_step_from_yaml()
  • STEP_TYPE_YAML_PARSERS
  • parse_step_from_yaml()
  • load_workflow_config()
  • Workflow Step Serialization
  • serialize_bash_step_to_yaml()
  • serialize_simulation_step_to_yaml()
  • serialize_pytest_step_to_yaml()
  • serialize_python_step_to_yaml()
  • serialize_notebook_step_to_yaml()
  • STEP_TYPE_YAML_SERIALIZERS
  • workflow_config_to_dict()
  • Workflow Step Validation
  • validate_bash_step()
  • validate_simulation_step()
  • validate_pytest_step()
  • validate_python_step()
  • validate_notebook_step()
  • psimulate
  • Branch and Keyspace Management
  • Keyspace
  • calculate_input_draws()
  • calculate_random_seeds()
  • calculate_keyspace()
  • load_branch_configuration()
  • expand_branch_templates()
  • validate_artifact_path()
  • psimulate CLI
  • Environment Variables
  • EnvVariable
  • psimulate Jobs
  • BackupConfiguration
  • generate_task_id()
  • JobParameters
  • build_job_parameters_from_keyspace()
  • build_job_list()
  • already_complete()
  • Model specification management
  • parse()
  • persist()
  • File Path Management
  • InputPaths
  • OutputPaths
  • transform_perf_df_for_appending()
  • append_child_job_data()
  • generate_runner_job_data()
  • append_perf_data_to_central_logs()
  • Reproducible Programming Environments
  • validate()
  • psimulate Runner
  • report_initial_status()
  • try_run_vipin()
  • write_backup_metadata()
  • write_configuration()
  • main()
  • Results Management
  • Results CLI Options
  • Results Writing
  • write_metadata()
  • write_task_results()
  • get_completed_task_ids()
  • collect_metadata()
  • psimulate Workers
  • Load Test Worker
  • get_psimulate_test_dict()
  • sleep_test()
  • large_results_test()
  • work_horse()
  • Jobmon Task Runner
  • parse_args()
  • main()
  • Vivarium Worker
  • ParallelSimulationContext
  • work_horse()
  • get_backup()
  • get_sim_from_backup()
  • initialize_new_sim()
  • run_simulation()
  • get_sim_results()
  • do_sim_epilogue()
  • remove_backups()
  • vipin - Vivarium Performance Log Parser
• Glossary