Jobmon

Jobmon is a Scientific Workflow Management system that simplifies running computational workflows on distributed computing systems. It provides:

• Easy-to-use Python and R APIs for defining workflows

• Centralized monitoring of jobs, including statuses and errors

• Automatic retries to protect against cluster failures

• Resource-aware retries that scale memory and runtime after failures

• Workflow resumes to continue from where you left off

• Fine-grained job dependencies including support for job arrays

• A web-based GUI for monitoring and debugging

Quick Links

• Getting Started - New to Jobmon? Start here with installation and your first workflow.

• User Guide - Learn about workflows, tasks, resources, and monitoring.

• Configuration - Configure Jobmon for your environment.

• Advanced Topics - Arrays, dynamic resources, troubleshooting, and more.

Table of Contents

Getting Started

• Getting Started
  • What You’ll Learn
  • Choose Your Path

User Guide

• User Guide
  • Core Concepts
  • Language Guides
  • Operations
• Core Concepts
  • Task
  • Node
  • Directed Acyclic Graph (DAG)
  • Edge
  • Example of Nodes, Tasks, DAGs, and their Args
  • Task Template
  • Workflow
  • Tool
  • Cluster Name
  • Compute Resources
  • Dependencies
  • Arguments
  • Analogy to Programming Languages
  • Abstract, Concrete, and Runtime Objects
• Workflows
  • Creating Workflows
  • Running Workflows
  • Resuming Workflows
  • Concurrency Control
  • See Also
• Tasks
  • Task Templates
  • Creating Tasks
  • Dependencies
  • Task Attributes
  • Retries
  • Checking Task Status
  • See Also
• Compute Resources
  • Basic Resources
  • Resource Hierarchy
  • YAML Configuration
  • Automatic Retries
  • Fallback Queues
  • Dynamic Resources
  • Checking Resource Usage
  • See Also
• Monitoring
  • Graphical User Interface (GUI)
  • Command Line Interface (CLI)
  • Programmatic Access
  • Task Statuses
  • See Also
• CLI Reference
  • Usage
  • Status Commands
  • Self-Service Commands
  • Utility Commands
  • See Also

Configuration

• Configuration
  • Overview
  • Configuration Files
  • Environment Variables
  • Sections
  • See Also
• Logging
  • Logging Guide

Advanced Topics

• Advanced Topics
  • Advanced Workflow Patterns
  • Operations & Maintenance
• Advanced Usage
  • Arrays
  • Retries
  • Resuming a Workflow
  • Fail Fast
  • Workflow Run Results and Exception Handling
  • Fallback Queues
  • Dynamic Task Resources
  • Advanced Task Dependencies
  • Concurrency Limiting
  • Jobmon Self-Service Commands
  • Resource Usage
  • Error Logs
  • Python Logging
• Task Generator
• Monitoring and Debugging
  • Graphical User Interface (GUI)
  • Jobmon Command Line Interface (CLI) Status Commands
  • Jobmon Database
  • Database Tables
  • Jobmon Statuses
• Troubleshooting
  • Quick Diagnostics
  • Common Errors
  • Resource Errors
  • Debugging Workflows
  • Common Patterns
  • Getting More Help
  • See Also
• Performance
  • Overview
  • Key Principles
  • Resource Optimization
  • Concurrency Limits
  • Monitoring Performance
  • Future Content
• Migration Guide
  • General Upgrade Process
  • Version History
  • Upgrade Guides
  • Common Migration Issues
  • Getting Help

IHME Users

• IHME Users
  • Getting Started at IHME
  • IHME Infrastructure

Reference

• Glossary of terms
  • For users
• API Reference
  • core
  • main
  • dummy
  • client
  • server
  • sequential
  • distributor
  • worker_node
  • multiprocess

Developer Guide

• Developer Guide
  • Getting Started
  • Development Workflow
  • Quick Commands
  • Repository Structure
  • Sections
  • See Also

Architecture

• Architecture and Design
  • Overview
  • Key Design Principles
  • Documentation Sections
  • Design Documents
  • See Also

Indices and Tables

• Index

• Module Index

• Search Page