Observability

How to see what the Colden kit is doing: where traces, metrics and logs go, how to follow one organization's request through the API, the outbox and the worker, and what never appears in telemetry.

Every Colden host sends traces, metrics and logs over OTLP. In Cloverheap's environments the endpoint is the Elastic stack; any OpenTelemetry Collector works the same way.

Turning it on

Set the endpoint, and the token when the endpoint needs one, as environment variables of the hosts (in a cluster, through the Kubernetes Secret and the chart's values):

Telemetry__OtlpEndpoint=https://apm.example.test:8200
Telemetry__Protocol=Grpc
Telemetry__OtlpHeaders="Authorization=Bearer <token>"   # a secret

With no endpoint nothing is exported; the hosts still log to the console.

Watching it locally

In Development, Telemetry__Exporter=Console prints every span, metric and log record to the console:

Telemetry__Exporter=Console dotnet run --project apps/api/Colden.Api
curl http://localhost:5080/      # the request span prints; health endpoints are not traced

The setting is ignored outside Development.

Following one request

Each span of a request carries the same trace id, from the API through the outbox and the PostgreSQL queue to the worker's handler. Filter by these attributes:

Attribute	Meaning
`tenant.id`	the organization; set on the request span when the API resolves the organization and on the handler span when the worker restores it
`user.id`	the signed-in user (from the `colden:user_id` claim)
`correlation.id`	the `X-Correlation-Id` request header or the request id, carried to handlers in the `colden.correlation` message header

service.name tells the hosts apart: colden-api, colden-worker, colden-migrate.

Operational metrics and the system screen

Every Diagnostics:ReportIntervalSeconds (30), while Diagnostics:Enabled (true; the test hosts turn it off), the worker runs its health checks and reads its jobs and queues. It writes the result into its row of worker_reports (one per instance, by host name) and exports it as gauges on the Colden meter:

Metric	Value
`colden.jobs.failed`, `colden.jobs.enqueued`, `colden.jobs.scheduled`, `colden.jobs.processing`	Hangfire's counts
`colden.messaging.queue.waiting` (tag `queue`)	The messages waiting in each queue
`colden.messaging.queue.dead_letters` (tag `queue`)	The failed messages in each queue's error queue

Super-admins read the same through the api (ADM-05):

GET /v1/admin/system/health returns every check of the api with its description and each worker instance's last report. A report older than Diagnostics:StaleAfterSeconds (120) is marked stale, and the overall status is Degraded while no worker reports.
GET /v1/admin/system/queues returns each queue's waiting, scheduled and failed messages and the latest failures.
POST /v1/admin/system/queues/{queue}/replay and .../discard (optionally { "messageType": "..." }) send a queue's failed messages back or delete them, logged as warnings.
GET /v1/admin/system/jobs returns the jobs of the freshest report.

What never appears

Exported log records pass through RedactionProcessor, which replaces email addresses and credentials (bearer tokens, Stripe keys and webhook secrets, the kit's API keys, Mailgun keys, Kratos session tokens). Code still must not log secrets or personal data: redaction is the safety net, not the rule.

Volume and sampling

The worker's message store polls PostgreSQL continuously; each poll is a short database span without a parent request. With Telemetry:SampleRatio at 1 every one of them is exported. In production, lower the ratio (for example 0.1) or filter parentless database spans in the collector; requests and the work they cause stay complete, because children follow their parent's sampling decision.