client.task_template ==================== .. py:module:: client.task_template .. autoapi-nested-parse:: A framework that many tasks have in common while varying by a declared set of arguments. Attributes ---------- .. autoapisummary:: client.task_template.logger Classes ------- .. autoapisummary:: client.task_template.TaskTemplate Module Contents --------------- .. py:data:: logger .. py:class:: TaskTemplate(template_name: str, requester: Optional[jobmon.core.requester.Requester] = None) Task Template outlines the structure of a Task to give it more context within the DAG. A Task Template defines a framework that many tasks have in common while varying by a declared set of arguments. Groups tasks of a type. Declares the concrete arguments that instances may vary over either from workflow to workflow or between nodes in the stage of a dag. :param template_name: the name of this task template. :param requester: object to communicate with the flask services. .. py:attribute:: template_name .. py:attribute:: requester :value: None .. py:method:: get_task_template(tool_version: jobmon.client.tool_version.ToolVersion, template_name: str) -> TaskTemplate :classmethod: Get a bound instance of TaskTemplate. :param tool_version: ToolVersion, to associate this task template with :param template_name: name of this specific task template .. py:method:: from_wire(wire_tuple: Tuple, tool_version: jobmon.client.tool_version.ToolVersion) -> TaskTemplate :classmethod: Get a bound instance of TaskTemplate from the http wire format. :param wire_tuple: Wire format for ToolVersion defined in jobmon.serializers. :param tool_version: ToolVersion, to associate this task template with :param requester: communicate with the flask services. .. py:method:: bind(tool_version: jobmon.client.tool_version.ToolVersion) -> None Bind task template to the db. :param tool_version: the ToolVersion this task template is associated with. .. py:property:: is_bound :type: bool If the task template version has been bound to the database. .. py:property:: id :type: int Unique id from db if task_template has been bound. .. py:property:: tool_version :type: jobmon.client.tool_version.ToolVersion The ToolVersion this task_template has been bound to. .. py:property:: task_template_versions :type: List[jobmon.client.task_template_version.TaskTemplateVersion] Version of task template if it has been bound. .. py:property:: active_task_template_version :type: jobmon.client.task_template_version.TaskTemplateVersion The TaskTemplateVersion to use when spawning tasks. .. py:property:: default_cluster_name :type: str Default cluster_name associated with active tool version. .. py:property:: default_max_attempts :type: Optional[int] Default max attempts of the active tool version. .. py:method:: set_default_max_attempts(value: int) -> None Set default max_attempts. :param value: value of max_attempts. .. py:property:: default_compute_resources_set :type: Dict[str, Dict[str, Any]] Default compute resources associated with active tool version. .. py:property:: default_resource_scales_set :type: Dict[str, Dict[str, float]] Default resource scales associated with active tool version. .. py:method:: update_default_compute_resources(cluster_name: str, **kwargs: Any) -> None Update default compute resources in place only overridding specified keys. If no default cluster is specified when this method is called, cluster_name will become the default cluster. :param cluster_name: name of cluster to modify default values for. :param \*\*kwargs: any key/value pair you want to update specified as an argument. .. py:method:: update_default_resource_scales(cluster_name: str, **kwargs: Any) -> None Update default resource scales in place only overridding specified keys. If no default cluster is specified when this method is called, cluster_name will become the default cluster. :param cluster_name: name of cluster to modify default values for. :param \*\*kwargs: any key/value pair you want to update specified as an argument. .. py:method:: set_default_compute_resources_from_yaml(yaml_file: str, default_cluster_name: str = '') -> None Set default ComputeResources from a user provided yaml file for task template level. :param default_cluster_name: name of cluster to set default values for. :param yaml_file: the yaml file that is providing the compute resource values. .. py:method:: set_default_resource_scales_from_yaml(yaml_file: str, default_cluster_name: str = '') -> None Set default Resource Scales from a user provided yaml file for task template level. :param default_cluster_name: name of cluster to set default values for. :param yaml_file: the yaml file that is providing the compute resource values. .. py:method:: set_default_compute_resources_from_dict(cluster_name: str, compute_resources: Dict[str, Any]) -> None Set default compute resources for a given cluster_name. If no default cluster is specified when this method is called, cluster_name will become the default cluster. :param cluster_name: name of cluster to set default values for. :param compute_resources: dictionary of default compute resources to run tasks with. Can be overridden at task level. dict of {resource_name: resource_value} .. py:method:: set_default_resource_scales_from_dict(cluster_name: str, resource_scales: Dict[str, float]) -> None Set default resource scales for a given cluster_name. If no default cluster is specified when this method is called, cluster_name will become the default cluster. :param cluster_name: name of cluster to set default values for. :param resource_scales: dictionary of default resource scales to adjust task resources with. Can be overridden at task level. .. py:method:: set_active_task_template_version_id(task_template_version_id: Union[str, int] = 'latest') -> None The TaskTemplateVersion that is set as the active one (latest is default). :param task_template_version_id: which version to set as active on this object. .. py:method:: set_active_task_template_version(task_template_version: jobmon.client.task_template_version.TaskTemplateVersion) -> None The TaskTemplateVersion that is set as the active one. :param task_template_version: which version to set as active on this object. .. py:method:: load_task_template_versions() -> None Load task template versions associated with this task template from the database. .. py:method:: get_task_template_version(command_template: str, node_args: Optional[List[str]] = None, task_args: Optional[List[str]] = None, op_args: Optional[List[str]] = None, default_cluster_name: str = '', default_compute_resources: Optional[Dict[str, Any]] = None, default_resource_scales: Optional[Dict[str, float]] = None, default_max_attempts: Optional[int] = None) -> jobmon.client.task_template_version.TaskTemplateVersion Create a task template version instance. If it already exists, activate it. :param command_template: an abstract command representing a task, where the arguments to the command have defined names but the values are not assigned. eg: '{python} {script} --data {data} --para {para} {verbose}' :param node_args: any named arguments in command_template that make the command unique within this template for a given workflow run. Generally these are arguments that can be parallelized over. :param task_args: any named arguments in command_template that make the command unique across workflows if the node args are the same as a previous workflow. Generally these are arguments about data moving though the task. :param op_args: any named arguments in command_template that can change without changing the identity of the task. Generally these are things like the task executable location or the verbosity of the script. :param default_cluster_name: the default cluster to run each task associated with this template on. :param default_compute_resources: dictionary of default compute resources to run tasks with. Can be overridden at task level. dict of {resource_name: resource_value}. Must specify default_cluster_name when this option is used. :param default_resource_scales: dictionary of default resource scales to adjust task resources with. Can be overridden at task level. dict of {resource_name: scale_factor}. Scale factor can be a numeric value, a Callable that will be applied to the existing resources, or an Iterator. Any Callable should take a single numeric value as its sole argument. Any Iterator should only yield numeric values. Any Iterable can be easily converted to an Iterator by using the iter() built-in (e.g. iter([80, 160, 190])). :param default_max_attempts: default max_attempts associated with this template on. .. py:method:: create_task(name: str = '', upstream_tasks: List[jobmon.client.task.Task] = [], task_attributes: Union[List, dict] = {}, max_attempts: Optional[int] = None, compute_resources: Optional[Dict[str, Any]] = None, compute_resources_callable: Optional[Callable] = None, resource_scales: Optional[Dict[str, Any]] = None, cluster_name: str = '', fallback_queues: Optional[List[str]] = None, **kwargs: Any) -> jobmon.client.task.Task Create an instance of a task associated with this template. :param name: a name associated with this specific task :param upstream_tasks: Task objects that must be run prior to this one :param task_attributes: attributes and their values or just the attributes that will be given values later :type task_attributes: dict or list :param max_attempts: Number of attempts to try this task before giving up. Default is 3. :param cluster_name: name of cluster to run task on. :param compute_resources: dictionary of default compute resources to run tasks with. Can be overridden at task template or task level. dict of {resource_name: resource_value} :param compute_resources_callable: compute resources generating callable. :param resource_scales: How much users want to scale their resource request if the the initial request fails. Scale factor can be a numeric value, a Callable that will be applied to the existing resources, or an Iterator. Any Callable should take a single numeric value as its sole argument. Any Iterator should only yield numeric values. Any Iterable can be easily converted to an Iterator by using the iter() built-in (e.g. iter([80, 160, 190])). :type resource_scales: dict :param \*\*kwargs: values for each argument specified in command_template :returns: ExecutableTask :raises ValueError: if the args that are supplied do not match the args in the command template. .. py:method:: create_tasks(max_attempts: Optional[int] = None, upstream_tasks: Optional[List[jobmon.client.task.Task]] = None, max_concurrently_running: int = MaxConcurrentlyRunning.MAXCONCURRENTLYRUNNING, compute_resources: Optional[Dict[str, Any]] = None, compute_resources_callable: Optional[Callable] = None, resource_scales: Optional[Dict[str, Any]] = None, cluster_name: str = '', name: Optional[str] = None, **kwargs: Any) -> List[jobmon.client.task.Task] Creates a set of tasks equal to the cross product of all node args. :param max_attempts: the max number of attempts a task in the array can be retried :param upstream_tasks: dependencies for all tasks in this array :param max_concurrently_running: the max number of tasks that can run at once :param compute_resources: resources to associate with this array, if different from the task template default resources :param compute_resources_callable: a function that can dynamically generate resources on retries, if different from the task template callable. :param resource_scales: Parameters for how aggressive we will rescale tasks that are resource killed. :param cluster_name: The cluster the array will run on :param \*\*kwargs: task, node, and op_args as defined in the command template. If you provide node_args as an iterable, they will be expanded. :param name: the name of the array. .. py:method:: resource_usage(workflows: Optional[List[int]] = None, node_args: Optional[Dict[str, Any]] = None, ci: Optional[float] = None) -> Optional[dict] Get the aggregate resource usage for a TaskTemplate.