Vivarium Cluster Tools Documentation

Vivarium cluster tools is a python package that makes running vivarium simulations at scale on a Slurm cluster easy.

This package is part of the vivarium-suite monorepo and sits alongside vivarium-engine (the simulation framework) and other vivarium-* libs.

• Installation
• Running simulations in parallel
  • Restarting a Simulation
  • Expanding a Simulation
• YAML Basics
  • Structure
  • Comments
  • Mappings
  • Lists
  • Composite Data
• The Branches File
  • Uncertainty
  • Configuration Variations
• Logging
  • Top-level logs
  • Cluster logs
  • Worker logs
  • Performance logs
• API Reference
  • Shared CLI tools
  • with_verbose_and_pdb()
  • with_sim_verbosity()
  • coerce_to_full_path()
  • pass_shared_options()
  • MinutesOrNone
  • load_run_config()
  • with_run_config()
  • resolve_deprecated_positional()
  • Logging Utilities
  • add_logging_sink()
  • configure_main_process_logging_to_terminal()
  • configure_main_process_logging_to_file()
  • vivarium.cluster_tools Utilities
  • get_cluster_name()
  • mkdir()
  • backoff_and_retry()
  • psimulate
  • Branch and Keyspace Management
  • Keyspace
  • calculate_input_draws()
  • calculate_random_seeds()
  • calculate_keyspace()
  • load_branch_configuration()
  • expand_branch_templates()
  • validate_artifact_path()
  • psimulate CLI
  • Environment Variables
  • EnvVariable
  • psimulate Jobs
  • generate_task_id()
  • JobParameters
  • build_job_list()
  • already_complete()
  • Model specification management
  • parse()
  • persist()
  • File Path Management
  • InputPaths
  • OutputPaths
  • transform_perf_df_for_appending()
  • append_child_job_data()
  • generate_runner_job_data()
  • append_perf_data_to_central_logs()
  • Reproducible Programming Environments
  • validate()
  • psimulate Runner
  • report_initial_status()
  • try_run_vipin()
  • write_backup_metadata()
  • write_configuration()
  • main()
  • Cluster Interaction
  • Cluster CLI options
  • with_queue_and_max_runtime()
  • Cluster Interface
  • validate_cluster_environment()
  • NativeSpecification
  • get_workflow_timeout_seconds()
  • Results Management
  • Results CLI Options
  • Results Writing
  • write_metadata()
  • write_task_results()
  • get_completed_task_ids()
  • collect_metadata()
  • psimulate Workers
  • Load Test Worker
  • get_psimulate_test_dict()
  • sleep_test()
  • large_results_test()
  • work_horse()
  • Jobmon Task Runner
  • parse_args()
  • main()
  • Vivarium Worker
  • ParallelSimulationContext
  • work_horse()
  • get_backup()
  • get_sim_from_backup()
  • initialize_new_sim()
  • run_simulation()
  • get_sim_results()
  • do_sim_epilogue()
  • remove_backups()
  • vipin - Vivarium Performance Log Parser
• Glossary