Testing

Jobmon uses pytest with nox for testing. Tests use SQLite databases that are automatically created for each test session (or worker when running in parallel).

Running Tests with Nox

Nox is the preferred way to run tests as it manages virtual environments automatically.

Run all tests:

nox -s tests -- tests/

Re-use the existing test environment (faster):

nox -r -s tests -- tests/

Run tests in parallel:

nox -r -s tests -- tests/ -n 4

Run a specific test file:

nox -r -s tests -- tests/integration/client/test_workflow.py

Run a single test:

nox -r -s tests -- tests/integration/client/test_workflow.py::test_workflow_bind

Test Database Setup

Tests use SQLite databases created automatically:

  • Each test session (or parallel worker) gets its own database

  • Databases are created in temporary directories

  • The db_engine fixture creates and migrates the database

  • The db_session fixture provides a transactional session per test

Key fixtures are defined in tests/fixtures/:

  • database.py - Database engine and session fixtures

  • server.py - Web server process and client connection fixtures

  • workflows.py - Tool, task template, and workflow fixtures

Test Organization

Tests are organized by test type:

Unit tests (tests/unit/):

  • tests/unit/core/ - Configuration, utilities, templates

  • tests/unit/client/ - Client library unit tests

  • tests/unit/server/ - Server logic unit tests

  • tests/unit/swarm/ - Swarm component unit tests (mocked)

  • tests/unit/logging/ - All logging configuration tests

Integration tests (tests/integration/):

  • tests/integration/client/ - Workflow/task binding, arrays

  • tests/integration/server/ - Database operations, routes

  • tests/integration/distributor/ - Task instantiation, triaging

  • tests/integration/swarm/ - Swarm execution integration

  • tests/integration/reaper/ - Workflow cleanup

  • tests/integration/cli/ - CLI command tests

End-to-end tests (tests/e2e/):

  • Full workflow execution tests with real distributors

Note

Unit tests run without a server and are fast (~16 seconds). Integration tests require a server and database. E2E tests exercise complete workflow execution.

Other Test Commands

Linting and formatting:

nox -s lint      # Run linters (flake8, mypy)
nox -s format    # Run formatters (black, isort)

Type checking:

nox -s typecheck

Generate ERD diagram:

nox -s schema_diagram

Load Testing

Load tests verify Jobmon’s performance under heavy usage. These are not automated and require manual execution.

To run a load test:

  1. Deploy the Jobmon version to test

  2. Configure the test parameters in deployment/tests/sample.yaml

  3. Run: python deployment/tests/multi_workflow_test.py --yaml_path deployment/tests/sample.yaml --scratch_dir {output_dir}

Smoke Testing

The six_job_test.py script is a quick smoke test that verifies basic system functionality:

python deployment/tests/six_job_test.py {cluster_name}

This confirms communication between client, server, and database is working.