feat(workflow): trace step attempts with OpenTelemetry

Merged
MA
8d ago
+584-12

Requested by Masen · project thread

First PR for ENG-13483 (plan items A, B, C).

Type of change

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update (READMEs of both packages updated here)

What and why

reflex-workflow had no OpenTelemetry code. Checked on main with the SDK, an in-memory exporter and ReflexInstrumentor().instrument():

  • a run started from a traced event handler produced steps that were invisible to the trace; anything a step traced itself started an unrelated trace;
  • every step task inherited whatever context was current when run_workflows was entered (asyncio.create_task copies contextvars for the runner loop and each step), so a worker entered inside a span parented every later step under that long-ended span.

This PR:

  • reflex_workflow.otel, on the reflex_base.otel pattern: an enabled flag the engine checks at each trace point, opentelemetry imports confined to enable(). reflex-workflow keeps its SQLAlchemy-only dependency list and imports nothing from opentelemetry until instrumented (a test pins that).
  • ReflexInstrumentor enables and disables the workflow trace points alongside the framework's when reflex_workflow.otel is importable, so one instrument() call, or opentelemetry-instrument, covers both.
  • One CONSUMER span per step attempt, named <table>.<step>, started from an explicit empty parent context so it is a trace root whatever the worker inherited. It links to the span that scheduled it rather than nesting under it: a step may run days after wake_in, on another worker after a lease expiry, or as one of many fanned-out children. Attributes: reflex.workflow.table, .step, .attempt, .lane, .outcome; error.type and the recorded exception on a failed attempt (status ERROR for retry/failed). Step arguments and row columns are never recorded.
  • Producer context carried in the row without a schema change. Call.encode() stamps the current W3C traceparent into the payload under a reserved top-level key of next_args, next to the claim's existing event_key, so start, run, deliver, chaining, schedules and fan_out all carry it. execute() strips it before the step sees its arguments; a retry re-stamps the failed attempt's span, so retries chain attempt → attempt.

Not in this PR (later items on the ticket): suppressing the engine's bookkeeping queries from DB instrumentations, and worker metrics.

Tests

  • tests/units/reflex_workflow/test_otel.py (no Postgres): disabled path imports no opentelemetry and changes nothing; stamp/encode carry the current span; the step span is a root even when entered under another span, with the link, kind and attributes; unusable traceparent values yield no link; failed attempts record the exception and status.
  • test_runner.py::test_steps_are_traced_as_linked_root_spans (Postgres): a chain started under a handler span links first → handler and second → first; a flaky step's three attempts are retry, retry, ok, each linked to the previous attempt; next_args is clean afterwards.
  • tests/units/reflex_otel/test_init.py: the instrumentor toggles the workflow trace points.

Ran locally: full tests/units/reflex_workflow against Postgres 16 plus tests/units/reflex_otel and tests/units/reflex_base/test_otel.py (1122 passed), ruff check/format, pyright on the touched packages and tests, codespell.

Generated by Claude Code