core.task_generator =================== .. py:module:: core.task_generator Attributes ---------- .. autoapisummary:: core.task_generator.SIMPLE_TYPES core.task_generator.BUILT_IN_COLLECTIONS core.task_generator.OPTIONAL_TYPES core.task_generator.TASK_RUNNER_NAME core.task_generator.TASK_RUNNER_SUB_COMMAND core.task_generator.HELP_TEXT_INTRO_FORMAT core.task_generator.SERIALIZED_EMPTY_STRING core.task_generator.logger Classes ------- .. autoapisummary:: core.task_generator.TaskGenerator core.task_generator.TaskGeneratorDocumenter core.task_generator.TaskGeneratorModuleDocumenter Functions --------- .. autoapisummary:: core.task_generator.create_task_name core.task_generator._is_multidimensional_type core.task_generator._find_executable_path core.task_generator._is_unannotated_built_in_collection_type core.task_generator._get_collection_type core.task_generator._get_generic_type_parameters core.task_generator._is_annotated_built_in_collection_type core.task_generator._zip_collection_items_and_types core.task_generator.is_optional_type core.task_generator.get_optional_type_parameter core.task_generator._clean_arg_name core.task_generator._get_short_description core.task_generator.make_cli_argument_string core.task_generator.task_generator core.task_generator.get_tasks_by_node_args core.task_generator._generate_nodes core.task_generator._format_description core.task_generator._format_options core.task_generator.setup Module Contents --------------- .. py:data:: SIMPLE_TYPES .. py:data:: BUILT_IN_COLLECTIONS .. py:data:: OPTIONAL_TYPES .. py:data:: TASK_RUNNER_NAME :value: 'worker_node_entry_point' .. py:data:: TASK_RUNNER_SUB_COMMAND :value: 'task_generator' .. py:data:: HELP_TEXT_INTRO_FORMAT :value: 'Command Line Documentation for {full_path}' .. py:data:: SERIALIZED_EMPTY_STRING :value: '""' .. py:data:: logger .. py:function:: create_task_name(kwargs_for_name: dict, prefix: str = '', name_func: Optional[Callable] = None) -> str Create a task name from the kwargs. .. py:function:: _is_multidimensional_type(type_hint: Any) -> bool .. py:function:: _find_executable_path(executable_name: str) -> str .. py:function:: _is_unannotated_built_in_collection_type(obj_type: Any) -> bool Return whether the ``obj_type`` is an unannotated, built-in collection type. Ex. ``list`` would return ``True``, ``tuple`` would return ``True``, ``list[str]`` would return ``False``. .. py:function:: _get_collection_type(obj_type: Any) -> Optional[Type] Return the type of collection. To check, we extract the ``__origin__`` component of the ``obj_type`` -- this would be the type of collection (list, tuple, set). Ex. ``list[int]`` would return ``list``, ``tuple[str]`` would return ``tuple``, ``float`` would return ``None`` (because it is not a collection). .. py:function:: _get_generic_type_parameters(obj_type: Any) -> Any Return the parameters for a generic type. The results are a tuple of types. For example, a ``list[int]`` returns ``(int,)``, a ``tuple[int, str]``, returns ``(int, str)``, and a ``Union[str, None]`` returns ``(str, NoneType)``. .. py:function:: _is_annotated_built_in_collection_type(obj_type: Any) -> bool Return whether the type is a properly annotated built-in collection type. .. py:function:: _zip_collection_items_and_types(obj: Collection[str], collection_items_type: tuple) -> Any Return a list of tuples mapping collection items and their types. Ex. ``obj`` is ``[1, 2, 3]`` and ``collection_items_type`` is ``(int,)`` would return ``[(1, int), (2, int), (3, int)]``. .. py:function:: is_optional_type(obj_type: Any) -> bool Return whether the ``obj_type`` is an Optional type. Ex. ``Optional[str]`` would return ``True``, ``NoneType`` would return ``True``, ``str | None`` would return ``True``, ``str`` would return ``False``. .. py:function:: get_optional_type_parameter(obj_type: Type) -> Type Return the type inside an optional. .. py:function:: _clean_arg_name(arg_name: str) -> str Remove the ``--`` from the arg name and convert dashes to underscores. .. py:function:: _get_short_description(task_function_docstring: docstring_parser.Docstring) -> str .. py:function:: make_cli_argument_string(arg_value: Union[str, List]) -> str Make a CLI argument string from an argument name and value. For string, return itself. For list, say ["a", "b", "c"], return string '[a,b,c]' .. py:class:: TaskGenerator(task_function: Callable, serializers: Dict, tool_name: str, naming_args: Optional[List[str]] = None, max_attempts: Optional[int] = None, module_source_path: Optional[str] = None, name_func: Optional[Callable] = None, default_cluster_name: str = '', default_compute_resources: Optional[Dict[str, Any]] = None, default_resource_scales: Optional[Dict[str, float]] = None, yaml_file: Optional[str] = None) Class for auto-generating jobmon tasks from a python function. Arguments on the task_function are inspected to generate the task template, and type annotations are used for serializing and deserializing arguments to strings passed on the command line. The TaskGenerator has built in serializers for: * str, int, bool, float * list, tuple, set of other serializable types * Optional types (and types that are equivalent to Optional types) of other serializable types Users can also supply serializers for custom types by providing a dictionary of type to a tuple of serialization and deserialization functions. Serializers must always turn objects into a single string. Lists of strings are represented on the command line as ``arg_name='[value1,value2]'``. .. note:: While lists, tuples and sets (and their ``Optional`` counterparts) can currently serialize empty collections, we can't currently serialize custom types as empty lists. .. py:attribute:: task_function .. py:attribute:: serializers .. py:attribute:: tool .. py:attribute:: max_attempts :value: None .. py:attribute:: mod_name :value: 'Uninferable' .. py:attribute:: name .. py:attribute:: full_path :value: 'Uninferable:Uninferable' .. py:attribute:: module_source_path :value: None .. py:attribute:: params .. py:attribute:: _naming_args :value: None .. py:attribute:: _task_template :value: None .. py:attribute:: _default_cluster_name :value: '' .. py:attribute:: _default_compute_resources :value: None .. py:attribute:: _default_resource_scales :value: None .. py:attribute:: _yaml_file :value: None .. py:attribute:: name_func :value: None .. py:method:: _validate_task_function() -> None Check that a task can be generated from the task_function. Currently checks: - All parameters have type annotations - All parameters have serializers .. py:method:: _is_valid_annotation(annotation: Type) -> bool Returns True if the annotation is valid, False otherwise. .. py:method:: _generate_task_template() -> None Generate and store the task template. .. py:method:: _is_valid_type(obj: Any, expected_type: Type) -> bool Check that the obj type matches the expected type. .. py:method:: serialize(obj: Any, expected_type: Type) -> str Serialize ``obj``, validating that it is actually of type ``expected_type``. .. py:method:: serialize_array(obj: Any, expected_type: Type) -> List Serialize obj into a list of serialized string. .. py:method:: deserialize(obj: str, obj_type: Type) -> Any Deserialize ``obj``. .. py:method:: _cluster_resource_check(cluster_name: str, compute_resources: Optional[Dict]) -> str Make sure a cluster name is available to use, and compute_resource is a dict. .. py:method:: create_task(cluster_name: str = '', compute_resources: Optional[Dict] = None, resource_scales: Optional[Dict[str, Any]] = None, upstream_tasks: List[jobmon.client.task.Task] = [], task_attributes: Union[List[str], Dict[str, Any]] = {}, **kwargs: Any) -> jobmon.client.task.Task Create a task for the task_function with the given kwargs. .. py:method:: create_tasks(cluster_name: str = '', compute_resources: Optional[Dict] = None, resource_scales: Optional[Dict[str, Any]] = None, upstream_tasks: Optional[List[jobmon.client.task.Task]] = None, **kwargs: Any) -> List[jobmon.client.task.Task] Create a task array for the task_function with the given kwargs. .. py:method:: run(args: List[str]) -> Any Run the task_function with the given args and return any result. .. py:method:: help() -> str Return help text for the task_function. .. py:method:: _get_param_names_and_descriptions(task_function_docstring_param_names_to_annotations: Dict, task_function_docstring_param_names_to_descriptions: Dict) -> List[str] .. py:function:: task_generator(serializers: Dict, tool_name: str, naming_args: Optional[List[str]] = None, max_attempts: Optional[int] = None, module_source_path: Optional[str] = None, name_func: Optional[Callable] = None, default_cluster_name: str = '', default_compute_resources: Optional[Dict[str, Any]] = None, default_resource_scales: Optional[Dict[str, float]] = None, yaml_file: Optional[str] = None) -> Callable Decorator for generating jobmon tasks from a python function. .. py:function:: get_tasks_by_node_args(workflow: jobmon.client.workflow.Workflow, task_generator: TaskGenerator, node_args_dict: dict[str, Any], error_on_empty: bool = True) -> list[jobmon.client.task.Task] Get the tasks of a TaskGenerator in a workflow that have the given node arguments. This method does some value serialization and formatting before handing the node argument string over to ``workflow.get_tasks_by_node_args``. .. py:class:: TaskGeneratorDocumenter(name, arguments, options, content, lineno, content_offset, block_text, state, state_machine) Bases: :py:obj:`docutils.parsers.rst.Directive` Directive for generating documentation for a single task generator. .. py:attribute:: required_arguments :value: 1 Number of required directive arguments. .. py:attribute:: optional_arguments :value: 1 Number of optional arguments after the required arguments. .. py:attribute:: final_argument_whitespace :value: True May the final argument contain whitespace? .. py:attribute:: option_spec Mapping of option names to validator functions. .. py:method:: run() -> list[docutils.nodes.Node] The function sphinx/docutils use to generate documentation for the directive. .. py:method:: _load_task_generator(task_generator_path: str, module_path: Optional[str] = None) -> TaskGenerator Load the task generator from the given path. .. py:class:: TaskGeneratorModuleDocumenter(name, arguments, options, content, lineno, content_offset, block_text, state, state_machine) Bases: :py:obj:`docutils.parsers.rst.Directive` Directive for generating documentation for all the task generators in a module. .. py:attribute:: required_arguments :value: 1 Number of required directive arguments. .. py:attribute:: optional_arguments :value: 1 Number of optional arguments after the required arguments. .. py:attribute:: final_argument_whitespace :value: True May the final argument contain whitespace? .. py:attribute:: option_spec Mapping of option names to validator functions. .. py:method:: run() -> list[docutils.nodes.Node] The function sphinx/docutils use to generate documentation for the directive. .. py:method:: _load_module_task_generators(module_name: str, module_path: Optional[str] = None) -> tuple[types.ModuleType, list[TaskGenerator]] Load all the task generators in a given module. .. py:method:: _generate_module_section(module_name: str, module: types.ModuleType, task_generators: Iterable[TaskGenerator]) -> docutils.nodes.Node Makes the docutils node for the module section. Includes the docstring for the module, and a list of task generator names. Doesn't include any task generator detailed documentation. .. py:function:: _generate_nodes(task_generator: TaskGenerator, state: Any) -> docutils.nodes.Node Makes a docutils node with the documentation for a task generator. Includes the docstring description for the task generator and all the arguments. .. py:function:: _format_description(task_generator: TaskGenerator, parsed_docstring: docstring_parser.Docstring) -> list[str] Format the description of the task generator into proper rST. .. py:function:: _format_options(task_generator: TaskGenerator, parsed_docstring: docstring_parser.Docstring) -> list[str] Format the options of the task generator into proper rST. .. py:function:: setup(app: sphinx.application.Sphinx) -> dict The function that registers the extension with sphinx.