multiprocess.multiproc_distributor
==================================

.. py:module:: multiprocess.multiproc_distributor

.. autoapi-nested-parse::

   Multiprocess executes tasks in parallel using a thread pool.



Attributes
----------

.. autoapisummary::

   multiprocess.multiproc_distributor.logger


Classes
-------

.. autoapisummary::

   multiprocess.multiproc_distributor.LimitedSizeDict
   multiprocess.multiproc_distributor.MultiprocessDistributor
   multiprocess.multiproc_distributor.MultiprocessWorkerNode


Module Contents
---------------

.. py:data:: logger

.. py:class:: LimitedSizeDict(*args: int, **kwds: int)

   Bases: :py:obj:`collections.OrderedDict`


   Dictionary for exit info.

   Initialization of LimitedSizeDict.


   .. py:attribute:: size_limit


.. py:class:: MultiprocessDistributor(cluster_name: str, parallelism: int = 3, *args: tuple, **kwargs: dict)

   Bases: :py:obj:`jobmon.core.cluster_protocol.ClusterDistributor`


   Executes tasks locally in parallel using a ThreadPoolExecutor.

   Each submitted task runs in a thread that spawns a subprocess via Popen.
   Shared memory dicts track processes, futures, exit codes, and errors.

   Initialization of the multiprocess distributor.

   :param cluster_name: the name of the cluster.
   :param parallelism: how many parallel jobs to distribute at a time.


   .. py:attribute:: started
      :value: False



   .. py:property:: worker_node_entry_point
      :type: str


      Path to jobmon worker_node_entry_point.


   .. py:property:: cluster_name
      :type: str


      Return the name of the cluster type.


   .. py:method:: start() -> None

      Start the thread pool executor.



   .. py:method:: stop() -> None

      Terminate all running tasks and shut down the executor.



   .. py:method:: submit_to_batch_distributor(command: str, name: str, requested_resources: Dict[str, Any]) -> str

      Submit a single task to the thread pool.



   .. py:method:: submit_array_to_batch_distributor(command: str, name: str, requested_resources: Dict[str, Any], array_length: int) -> Dict[int, str]

      Submit an array task to the thread pool.

      Return: a mapping of array_step_id to distributor_id.



   .. py:method:: get_submitted_or_running(distributor_ids: Optional[List[str]] = None) -> Set[str]

      Get tasks that are active.



   .. py:method:: terminate_task_instances(distributor_ids: List[str]) -> None

      Terminate task instances.

      Kills subprocesses and cancels futures for the given IDs.

      :param distributor_ids: A list of distributor IDs.



   .. py:method:: get_queueing_errors(distributor_ids: List[str]) -> Dict[str, str]

      Get the task instances that have errored during queueing.



   .. py:method:: get_remote_exit_info(distributor_id: str) -> jobmon.core.exit_info.RemoteExitInfo

      Get the exit info about the task instance once done.



.. py:class:: MultiprocessWorkerNode

   Bases: :py:obj:`jobmon.core.cluster_protocol.ClusterWorkerNode`


   Task instance info for the Multiprocess distributor.

   Initialization of the multiprocess distributor worker node.


   .. py:property:: distributor_id
      :type: Optional[str]


      The id from the distributor.


   .. py:method:: get_exit_info(exit_code: int, error_msg: str) -> Tuple[str, str]

      Exit code and message.



   .. py:method:: get_usage_stats() -> Dict

      Usage information specific to the distributor.



   .. py:method:: initialize_logfile(log_type: str, log_dir: str, name: str) -> str

      Error and exit code info from the executor.



   .. py:property:: array_step_id
      :type: Optional[int]


      Return array_step_id .