OpenTelemetry Tracing¶

Helmfile can export OpenTelemetry traces of a run, which is especially useful in CI/CD: see where time is spent, which helm invocations dominate, how label selectors fan out, and what to target for improvement.

Tracing is experimental and off by default. It currently emits the command-level root span, state-loading spans (discover/load/render/parse), one span per external process (helm invocations, hooks, plugin execs), per-hook spans, and per-release spans for the sync/diff/delete/status/test/prepare paths.

Enabling¶

helmfile --otel-tracing -e production -l tier=backend apply

or via environment variable (handy for CI):

export HELMFILE_OTEL_TRACING=true
helmfile sync

--otel-tracing overrides HELMFILE_OTEL_TRACING when specified.

Configuration¶

Everything beyond the on/off switch uses the standard OpenTelemetry environment variables; helmfile defines no telemetry-specific variables of its own.

Variable	Default	Notes
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://localhost:4318`	OTLP endpoint; use `http://host:4317` for gRPC
`OTEL_EXPORTER_OTLP_PROTOCOL`	`http/protobuf`	or `grpc`
`OTEL_EXPORTER_OTLP_HEADERS`	—	e.g. `authorization=Bearer <token>`
`OTEL_TRACES_EXPORTER`	`otlp`	`otlp` \| `console` \| `none`
`OTEL_METRICS_EXPORTER`	`otlp`	`otlp` \| `console` \| `prometheus` \| `none` — see Metrics
`OTEL_TRACES_SAMPLER` (+ `OTEL_TRACES_SAMPLER_ARG`)	`parentbased_always_on`
`OTEL_SERVICE_NAME`	`helmfile`
`OTEL_RESOURCE_ATTRIBUTES`	—	e.g. `cicd.pipeline=deploy,cicd.run_id=4821`
`OTEL_PROPAGATORS`	`tracecontext,baggage`
`OTEL_SDK_DISABLED`	`false`	standard kill switch

HTTP proxies (HTTPS_PROXY/HTTP_PROXY) are honored automatically by the Go runtime.

Quick start without a collector¶

To inspect what would be exported, print spans as JSON on stdout:

OTEL_TRACES_EXPORTER=console helmfile --otel-tracing -l name=myrelease template

Quick start with Jaeger¶

docker run --rm -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one:latest

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
  helmfile --otel-tracing -e production apply
# Open http://localhost:16686 and pick service "helmfile"

What gets traced¶

The root span is named after the command (helmfile apply, helmfile sync, …) and carries helmfile.command, helmfile.file, helmfile.environment, and helmfile.selectors attributes, plus helmfile.exit_code and the error (if any) at the end.

Every external process helmfile starts — each helm invocation, hook command, and plugin exec — gets its own span nested under the command span: helm.exec (with helm.subcommand) for helm binaries, os.exec otherwise, both carrying exec.command, redacted exec.args, and exec.exit_code on failure.

State loading is traced too: helmfile.discover_states, one helmfile.load per state file (including nested helmfiles), with helmfile.render and helmfile.parse children — rendering is frequently the hidden time sink. Each hook execution produces a helmfile.hook span (with hook.event and hook.name) that its subprocess span nests under — and release-scoped hooks (presync/postsync & co. within sync/diff/delete) nest under their release’s span, keeping full release attribution.

Per-release spans (helmfile.release.sync / .diff / .delete / .status / .test / .prepare) carry the release name, namespace, chart, and labels, and nest under the state-file load span — with the release’s helm subprocesses (upgrade, diff, delete, status, test) nested under the release span. The remaining loops (flag preparation, template/lint/unittest) are being added incrementally; see the design proposal for the full taxonomy.

Secret-bearing command arguments (--set, --set-file, --username, --password, …) are never recorded in span attributes — they are masked before export.

Metrics¶

Tracing and metrics share the same switch and resource; metrics are exported through the standard metrics environment variables:

Variable	Default	Notes
`OTEL_METRICS_EXPORTER`	`otlp`	`otlp` \| `console` \| `prometheus` \| `none`
`OTEL_METRIC_EXPORT_INTERVAL`	`60000` (ms)	periodic export interval; the final flush happens at exit

Three instruments are emitted:

Metric	Type	Attributes
`helmfile.helm.exec.duration`	histogram (seconds, buckets tuned for 5ms–600s helm invocations)	`subcommand`, `success`
`helmfile.release.duration`	histogram (seconds, same buckets)	`verb`, `result` — plus `helmfile.release` and `helmfile.namespace` when per-release metrics are enabled (below)
`helmfile.release.count`	counter (unit `{release}`)	`verb` (sync/diff/delete/status/test/prepare), `result` (success/error)

Per-release metrics (opt-in, high cardinality)¶

By default metric dimensions are bounded enumerations — release names never appear, so time-series count stays constant regardless of fleet size. If you need per-release durations on dashboards, opt in:

HELMFILE_OTEL_METRICS_PER_RELEASE=true helmfile --otel-tracing sync

This adds helmfile.release and helmfile.namespace to helmfile.release.duration. Time-series count then scales with your release fleet: well-suited to bounded CI runs, but long-lived centralized collection needs a backend capacity/TTL story. (Per-release timing is always available in traces, without this flag.)

Both dimensions are bounded enumerations — never release names — so metric cardinality stays constant regardless of fleet size. The instrumentation scope (helmfile) carries the helmfile version.

Inspect them without a collector with OTEL_METRICS_EXPORTER=console.

CI trace correlation¶

If your CI system injects a W3C TRACEPARENT environment variable into jobs, helmfile joins that trace automatically, so helmfile spans appear inside your pipeline’s trace.

Troubleshooting¶

No spans arrive: check OTEL_EXPORTER_OTLP_ENDPOINT/protocol match your collector (4318 for http/protobuf, 4317 for grpc); verify with OTEL_TRACES_EXPORTER=console.
Tracing breaks nothing: exporter/sampler misconfiguration and export failures never fail a helmfile run; at worst you get no spans.
Still stuck? Set OTEL_SDK_DISABLED=false explicitly and rerun with --log-level debug.