core.otlp ========= .. py:module:: core.otlp .. autoapi-nested-parse:: Elegant OTLP integration for jobmon that prevents global log pollution. Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/core/otlp/formatters/index /autoapi/core/otlp/handlers/index /autoapi/core/otlp/manager/index /autoapi/core/otlp/resources/index /autoapi/core/otlp/utils/index Attributes ---------- .. autoapisummary:: core.otlp.OTLP_AVAILABLE core.otlp.JobmonOTLPStructlogHandler core.otlp.OpenTelemetryLogFormatter Classes ------- .. autoapisummary:: core.otlp.JobmonOTLPFormatter core.otlp.JobmonOTLPLoggingHandler core.otlp.JobmonOTLPManager Functions --------- .. autoapisummary:: core.otlp.create_log_exporter core.otlp.get_logger core.otlp.get_shared_logger_provider core.otlp.initialize_jobmon_otlp core.otlp.otlp_flush_on_exit core.otlp.validate_otlp_exporter_config core.otlp.add_span_details_processor core.otlp.get_current_span_details Package Contents ---------------- .. py:data:: OTLP_AVAILABLE :value: True .. py:class:: JobmonOTLPFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None) Bases: :py:obj:`logging.Formatter` Formatter that adds OpenTelemetry span details to jobmon logs. Note: For handlers using _JobmonOTLPLoggingHandler, this formatter is not used for OTLP output (handler creates OTLPLogRecord directly). It only affects console/file output. Initialize the formatter with specified format strings. Initialize the formatter either with the specified format string, or a default as described above. Allow for specialized date formatting with the optional datefmt argument. If datefmt is omitted, you get an ISO8601-like (or RFC 3339-like) format. Use a style parameter of '%', '{' or '$' to specify that you want to use one of %-formatting, :meth:`str.format` (``{}``) formatting or :class:`string.Template` formatting in your format string. .. versionchanged:: 3.2 Added the ``style`` parameter. .. py:method:: format(record: logging.LogRecord) -> str Format log record with OpenTelemetry span details. .. py:class:: JobmonOTLPLoggingHandler(level: int = logging.NOTSET, exporter: Optional[Union[Any, Dict]] = None, logger_provider: Optional[Any] = None) Bases: :py:obj:`logging.Handler` Universal OTLP logging handler with lazy initialization and attribute extraction. This handler works with both stdlib logging and structlog. It extracts structured attributes from structlog's thread-local ``event_dict`` when available, enabling rich OTLP exports with queryable fields. Note: ``JobmonOTLPStructlogHandler`` is an alias for this class (kept for backward compatibility). Both names can be used interchangeably in configuration. Supports flexible configuration patterns: 1. Inline dict configuration (server pattern):: handlers: otlp_logs: class: jobmon.core.otlp.JobmonOTLPLoggingHandler level: INFO exporter: module: opentelemetry.exporter.otlp.proto.grpc._log_exporter class: OTLPLogExporter endpoint: otelcol.dev.aks.scicomp.ihme.washington.edu:443 options: [["grpc.max_send_message_length", 16777216]] max_export_batch_size: 8 2. Pre-configured exporter instance:: handler = JobmonOTLPLoggingHandler(exporter=my_exporter) 3. Direct use with logger_provider (for testing):: handler = JobmonOTLPLoggingHandler(logger_provider=my_provider) Initialize with optional exporter config or pre-configured logger provider. :param level: Logging level for this handler :param exporter: Either a dict configuration or pre-configured OTLP exporter instance :param logger_provider: Optional pre-configured logger provider (for testing/direct use) .. py:method:: close() -> None Tidy up any resources used by the handler. This version removes the handler from an internal map of handlers, _handlers, which is used for handler lookup by name. Subclasses should ensure that this gets called from overridden close() methods. .. py:method:: emit(record: logging.LogRecord) -> None Emit log record to OTLP with extracted attributes. .. py:data:: JobmonOTLPStructlogHandler .. py:class:: JobmonOTLPManager OTLP manager for shared trace and log resources. This manager handles: - Trace provider setup (for distributed tracing) - Logger provider setup (for OTLP log export) - Resource detection (shared across components) - Request instrumentation (shared utility) All OTLP handlers should use the shared logger provider to avoid duplicate connections and log emissions. Initialize the OTLP manager. .. py:attribute:: tracer_provider :type: Optional[Any] :value: None .. py:attribute:: logger_provider :type: Optional[Any] :value: None .. py:method:: get_instance() -> JobmonOTLPManager :classmethod: Get or create the singleton OTLP manager with thread safety. .. py:method:: initialize() -> None Initialize trace and log providers with jobmon resources. .. py:method:: get_tracer(name: str) -> Optional[Any] Get a tracer for distributed tracing. .. py:method:: instrument_requests() -> None :classmethod: Instrument requests library for HTTP tracing. .. py:method:: shutdown() -> None Shutdown trace and log providers. .. py:method:: flush_and_shutdown() -> None Flush pending OTLP telemetry and shut down providers. .. py:function:: create_log_exporter(**kwargs: Any) -> Optional[Any] Create a pre-configured log exporter for client applications. This factory function creates exporters that can be passed to JobmonOTLPLoggingHandler for pure separation. :param \*\*kwargs: Exporter configuration (endpoint, headers, etc.) :returns: Pre-configured OTLP log exporter, or None if unavailable Example:: exporter = create_log_exporter( endpoint="otelcol.dev.aks:443", max_batch_size=8 ) handler = JobmonOTLPLoggingHandler(exporter=exporter) .. py:function:: get_logger(name: str) -> Optional[Any] Get a logger from the shared provider. This is the cleanest way for handlers to get OTLP loggers without dealing with manager instances or initialization complexity. :param name: Logger name (typically __name__) :returns: OTLP logger instance, or None if unavailable .. py:function:: get_shared_logger_provider() -> Optional[Any] Get the shared logger provider, initializing if needed. This function provides a clean interface for handlers to access the shared LoggerProvider without dealing with manager instances directly. Uses double-checked locking to prevent race conditions in multi-threaded environments like Kubernetes with multiple workers. :returns: The shared LoggerProvider instance, or None if unavailable .. py:function:: initialize_jobmon_otlp() -> JobmonOTLPManager Initialize OTLP for shared resources (traces and logs). This creates shared TracerProvider and LoggerProvider instances that should be used by all OTLP handlers to avoid duplicate connections. :returns: The OTLP manager instance with shared providers .. py:function:: otlp_flush_on_exit() -> Generator[Optional[JobmonOTLPManager], None, None] Context manager that guarantees OTLP flush when exiting. .. py:function:: validate_otlp_exporter_config(config: Any, exporter_type: str = 'log') -> list[str] Validate OTLP exporter configuration and return list of issues. :param config: Exporter configuration dictionary :param exporter_type: Type of exporter ('log', 'trace', 'metric') :returns: List of validation error messages. Empty list if valid. .. py:function:: add_span_details_processor(logger: Any, method_name: str, event_dict: Dict[str, Any]) -> Dict[str, Any] Structlog processor to add OpenTelemetry span details to log entries. :param logger: The logger instance (not used, but required by Structlog processor signature). :param method_name: The logging method name (e.g., "info", "debug"). :param event_dict: The event dictionary representing the log entry. :returns: The modified event dictionary with OpenTelemetry span details added. .. py:function:: get_current_span_details() -> Tuple[Optional[str], Optional[str], Optional[str]] Get details of the current OpenTelemetry span. :returns: Tuple of (span_id, trace_id, parent_span_id) as hex strings, or (None, None, None) .. py:data:: OpenTelemetryLogFormatter