APM, traces and the service map
Orbit Pulse turns the spans your services send into a service catalog, a trace explorer and a service map. This page is for developers who read traces and for project administrators who decide which traces to keep.
Spans arrive over OpenTelemetry. See Send telemetry with OpenTelemetry for the endpoints and for how Pulse reads span fields.
Find a slow or failing service
Section titled “Find a slow or failing service”Open APM to see the service catalog. Each row is a service in an environment, with its request rate, error rate and p50, p95 and p99 latency, a sparkline and sorting. You can filter by environment.
The numbers come from entry spans, the first span a service handles for a request. Percentiles are estimates from per-minute latency sketches.
Click a service to open its page:
- latency, throughput and error charts;
- a table of endpoints (the span resource, such as
GET /orders/{id}); - the latency distribution;
- the versions seen;
- the dependencies the service calls, such as databases and other services.
Choosing an endpoint focuses the totals and charts on it.
Search spans
Section titled “Search spans”The trace explorer uses the same query syntax as logs. You can filter by service, resource, duration, status and tags, and you get a histogram and facets.
service:checkout status:error @duration:>500msSee the query syntax reference for the span fields you can use.
Follow one request in the waterfall
Section titled “Follow one request in the waterfall”Open a trace to see its waterfall. Each bar is a span, with its details, attributes, events and the error stack when there is one. A trace shows up to 10,000 spans.
Logs and traces meet through the trace id, in both directions. From a trace you open the logs that carry its id, and from a log you open its trace. Use the same service.name in logs and traces so they meet under one service.
Read the service map
Section titled “Read the service map”The service map is built from client and server spans, and from producer and consumer spans. Each node is a service, a database, a queue or an external host. Each edge shows requests, errors and latency between two nodes.
The map covers at most the last 24 hours. When a window holds more than 200,000 client spans, Pulse samples traces by id and scales the counts.
Choose which traces to keep
Section titled “Choose which traces to keep”Pulse stores every span when it arrives and writes request, error and latency rollups at the same time. So service figures and the latency distribution count every request, whatever you keep.
A project administrator can thin out the rest in the Pulse Settings, under Trace retention:
| Setting | Meaning | Default |
|---|---|---|
| Share of traces to keep | From 0.1% to 100%. Applies to traces that are neither slow nor failing. | 100%, so nothing is removed |
| Slow threshold | A trace at least this long is always kept. | 1,000 ms |
Fifteen minutes after a trace ends, Pulse keeps it whole when any of these is true:
- It has an error span.
- An error event, from a span or an error log, carries its trace id.
- It lasted at least the slow threshold.
- Its trace id falls into the sampled share.
The spans of the other traces are deleted and cannot be restored. A trace removed this way answers “trace not retained” with the rule that removed it. Spans that arrive after a trace was decided are kept.
A changed setting applies to traces that end after you save it. Under the form, Pulse counts each day’s decisions: traces kept with errors, kept as slow, kept by sampling, and dropped traces and spans.