Tracing
Tracing tracks the progression of a single user request as it is handled by Pomerium.
Each unit of work is called a Span in a trace. Spans include metadata about the work, including the time spent in the step (latency), status, time events, attributes, links. You can use tracing to debug errors and latency issues in your applications, including in downstream connections.
Shared Tracing Settings
Config Key | Description | Required |
---|---|---|
tracing_provider | The name of the tracing provider. (e.g. jaeger, zipkin or datadog) | ✅ |
tracing_sample_rate | Percentage of requests to sample in decimal notation. Default is 0.0001 , or .01% | ❌ |
Set tracing_sample_rate = 1
if you want to see all requests in the tracings.
Datadog
Datadog is a real-time monitoring system that supports distributed tracing and monitoring.
Config Key | Description | Required |
---|---|---|
tracing_datadog_address | host:port address of the Datadog Trace Agent. Defaults to localhost:8126 | ❌ |
Jaeger (partial)
Warning At this time, Jaeger protocol does not capture spans inside the Proxy service. Please use Zipkin protocol with Jaeger for full support.
Jaeger is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and troubleshooting microservices-based distributed systems, including:
- Distributed context propagation
- Distributed transaction monitoring
- Root cause analysis
- Service dependency analysis
- Performance / latency optimization
Config Key | Description | Required |
---|---|---|
tracing_jaeger_collector_endpoint | Url to the Jaeger HTTP Thrift collector. | ✅ |
tracing_jaeger_agent_endpoint | Send spans to jaeger-agent at this address. | ✅ |
For quick local testing, use Jaeger all-in-one, which is an executable designed to launch the Jaeger UI, jaeger-collector, jaeger-query, and jaeger-agent, with an in-memory storage component.
docker run -d --name jaeger \
-e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
-e COLLECTOR_OTLP_ENABLED=true \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 4317:4317 \
-p 4318:4318 \
-p 14250:14250 \
-p 14268:14268 \
-p 14269:14269 \
-p 9411:9411 \
jaegertracing/all-in-one:1.45
Pomerium settings
tracing_provider: jaeger
tracing_jaeger_collector_endpoint: http://localhost:14268/api/traces
tracing_jaeger_agent_endpoint: http://localhost:6831
Open Jaeger UI at http://localhost:16686
in the browser to view Pomerium traces.
Zipkin
Zipkin is an open source distributed tracing system and protocol.
Many tracing backends support zipkin either directly or through intermediary agents, including Jaeger. For full tracing support, we recommend using the Zipkin tracing protocol.
Config Key | Description | Required |
---|---|---|
tracing_zipkin_endpoint | Url to the Zipkin HTTP endpoint. | ✅ |
Add next configuration for running zipkin
.
docker-compose.yaml
services:
zipkin:
image: openzipkin/zipkin:latest
ports:
- 9411:9411
Pomerium settings
tracing_provider: zipkin
tracing_zipkin_endpoint: http://localhost:9411/api/v2/spans
Open Zipkin UI at http://localhost:9411
in the browser to view Pomerium traces.
Remember to replace localhost
with the appropriate hostname or IP address for your specific setup.
If you are running Pomerium and Zipkin/Jaeger in separate Docker containers and they are on the same host, try using host.docker.internal
as the hostname instead of localhost
in the Pomerium settings. This allows containers to access services running on the host machine. Update the Zipkin/Jaeger endpoint URL in Pomerium accordingly.