multiprocess.multiproc_distributor

Multiprocess executes tasks in parallel using a thread pool.

Attributes

logger

Classes

LimitedSizeDict	Dictionary for exit info.
MultiprocessDistributor	Executes tasks locally in parallel using a ThreadPoolExecutor.
MultiprocessWorkerNode	Task instance info for the Multiprocess distributor.

Module Contents

multiprocess.multiproc_distributor.logger

class multiprocess.multiproc_distributor.LimitedSizeDict(*args: int, **kwds: int)

Bases: collections.OrderedDict

Dictionary for exit info.

Initialization of LimitedSizeDict.

size_limit

class multiprocess.multiproc_distributor.MultiprocessDistributor(cluster_name: str, parallelism: int = 3, *args: tuple, **kwargs: dict)

Bases: jobmon.core.cluster_protocol.ClusterDistributor

Executes tasks locally in parallel using a ThreadPoolExecutor.

Each submitted task runs in a thread that spawns a subprocess via Popen. Shared memory dicts track processes, futures, exit codes, and errors.

Initialization of the multiprocess distributor.

Parameters:

• cluster_name – the name of the cluster.

• parallelism – how many parallel jobs to distribute at a time.

started = False

property worker_node_entry_point: str

Path to jobmon worker_node_entry_point.

property cluster_name: str

Return the name of the cluster type.

start() → None

Start the thread pool executor.

stop() → None

Terminate all running tasks and shut down the executor.

submit_to_batch_distributor(command: str, name: str, requested_resources: Dict[str, Any]) → str

Submit a single task to the thread pool.

submit_array_to_batch_distributor(command: str, name: str, requested_resources: Dict[str, Any], array_length: int) → Dict[int, str]

Submit an array task to the thread pool.

Return: a mapping of array_step_id to distributor_id.

get_submitted_or_running(distributor_ids: List[str] | None = None) → Set[str]

Get tasks that are active.

terminate_task_instances(distributor_ids: List[str]) → None

Terminate task instances.

Kills subprocesses and cancels futures for the given IDs.

Parameters:

distributor_ids – A list of distributor IDs.

get_queueing_errors(distributor_ids: List[str]) → Dict[str, str]

Get the task instances that have errored during queueing.

get_remote_exit_info(distributor_id: str) → jobmon.core.exit_info.RemoteExitInfo

Get the exit info about the task instance once done.

class multiprocess.multiproc_distributor.MultiprocessWorkerNode

Bases: jobmon.core.cluster_protocol.ClusterWorkerNode

Task instance info for the Multiprocess distributor.

Initialization of the multiprocess distributor worker node.

property distributor_id: str | None

The id from the distributor.

get_exit_info(exit_code: int, error_msg: str) → Tuple[str, str]

Exit code and message.

get_usage_stats() → Dict

Usage information specific to the distributor.

initialize_logfile(log_type: str, log_dir: str, name: str) → str

Error and exit code info from the executor.

property array_step_id: int | None

Return array_step_id .