How Pulse stores telemetry: OpenTelemetry in, ClickHouse underneath
How Pulse takes in logs, traces and metrics over OTLP/HTTP, keeps them in ClickHouse with per-project retention and rollups, and why it has one query syntax.
Observability data has an awkward shape. It arrives in bursts from many senders, it is written once and never edited, and people read it in two very different ways: a live search for one needle, and an aggregate over millions of rows. This post walks through how Orbit Pulse handles that, from the moment an SDK sends a request to the moment a chart or a search reads it back.
Pulse is part of Orbit, which is pre-launch and in private preview. Everything described here is built and tested; where something is planned, we say so.
In: standard OpenTelemetry, no agent of our ownLink to section: In: standard OpenTelemetry, no agent of our own
Pulse takes in logs, traces and metrics over OTLP/HTTP, the standard OpenTelemetry protocol. You use the OpenTelemetry SDK and exporter for your language and point it at your company address; there is no proprietary agent to install. Each signal has its own endpoint under /otlp/v1/: logs, traces and metrics.
A request body is either application/x-protobuf or application/json, and the Content-Encoding can be gzip, deflate, zstd or br. The sender authenticates with a project ingest key, in an X-Pulse-Key header or as a bearer token. The key decides the project, and anything in the payload that claims a project is ignored.
Intake is strict about size and forgiving about content. A request carries at most 10,000 records. Each key has a records-per-minute budget that is all-or-nothing per request, so an over-budget request gets a 429 with Retry-After and stores nothing, rather than half of a batch. Inside an accepted request, a bad record does not sink the good ones. Pulse answers 200 with the standard partialSuccess object, keeps the valid records, and counts the rejected ones: logs outside the project’s retention window, spans without a trace or span id, metric points over a series limit.
The standard OpenTelemetry environment variables are all a service needs:
OTEL_EXPORTER_OTLP_ENDPOINT=https://acme.orbit.inortix.com/otlpOTEL_EXPORTER_OTLP_HEADERS=X-Pulse-Key=<ingest key>OTEL_SERVICE_NAME=checkoutResource attributes are mapped to the fields people filter on every day: service.name becomes the service, service.version the version, deployment.environment.name the environment and host.name the host. All resource attributes are also kept as tags, so nothing the SDK sent is thrown away.
Durable before it is acknowledgedLink to section: Durable before it is acknowledged
The most important line in the write path is a promise: when Pulse tells a sender 200, the data is safe. Pulse writes to ClickHouse in batches using its asynchronous insert mode, and it waits for the insert to be flushed before it answers. An acknowledged request is therefore durable without a message queue sitting in front of the database.
That design has a trade-off, and we measured it. In a development test, a load generator sent 649,000 log records in about a minute through gzip-compressed OTLP, with none failed or rejected, and the database held exactly that many rows afterwards. Throughput was bounded by the requests in flight multiplied by the flush wait, not by CPU, so sending more means more concurrent senders or larger batches. That was a test on a development stack, not a statement of capacity.
Underneath: ClickHouse, one table design per signalLink to section: Underneath: ClickHouse, one table design per signal
Telemetry lives in ClickHouse, a column-oriented database built for large append-only data. Logs, spans and metric points are stored in MergeTree tables with the same basic layout:
- Partitioned by day. A day is the unit that is created, queried and dropped. Reading “the last hour” touches one or two partitions, not the whole history.
- Ordered by project, service and time. Rows from the same project and service sit next to each other on disk, so a typical query, one service in one project over a time range, reads a narrow slice instead of scanning everything.
- A time-to-live on every row. Expiry is part of the table, not a cleanup job that has to be remembered.
The control data is kept elsewhere. PostgreSQL, shared with the rest of Orbit, holds what needs transactions and relations: ingest keys, saved views, monitors and their state, error groups, incidents and the links to Orbit work. Redis holds the ingest-key cache, rate limits and the pub/sub channel behind live tail. Live tail is a convenience over durable rows, never the only copy of anything. If Redis loses a message, the data is still in ClickHouse.
Retention per projectLink to section: Retention per project
Because every row carries its own expiry, retention is a per-project setting rather than a property of a table. A project administrator picks how long that project keeps its data, within the limit of the organization’s plan, and ClickHouse expires the rows on that schedule. A project that needs three days and a project that needs a month can share the same tables.
Retention is also enforced at the door: a record outside the project’s retention window is rejected at intake and counted in partialSuccess, so replayed old data cannot fill storage with rows that would expire at once. The limits per plan are on the pricing page.
Rollups: answers that do not depend on raw rowsLink to section: Rollups: answers that do not depend on raw rows
A dashboard that asks “how many requests, how many errors, and what is the 95th percentile latency for this service?” should not scan every span. Pulse computes those answers as it ingests. Entry spans (the spans that start a request in a service) and client spans are rolled up per minute with ClickHouse materialized views: hits, errors, duration and a latency sketch. Percentiles come from DDSketch states, a data structure that merges cleanly across minutes, services and time ranges while keeping a bounded relative error.
This matters most for a decision that otherwise loses information: sampling. Pulse keeps every error trace and every slow trace, and keeps the rest at a configured rate; a project administrator sets the rate and the slow threshold. The rollups are written at intake, before any sampling decision, so request counts, error counts and latency figures stay complete even when most of the fast, healthy traces are dropped later. The sampling is visible in the interface rather than hidden: a trace that was not kept answers with the rule that dropped it.
Metrics follow the same idea with a different limit. Each series, a name plus its tags, is recorded in a catalog, and the number of new series per project per day is capped. A tag with unbounded values, such as a user id, is the classic way to flood a metrics store; the cap turns that mistake into a visible rejection.
One query syntaxLink to section: One query syntax
Logs, spans and errors have different fields, but Pulse gives them one query syntax. service:checkout status:error means the same thing in log search, in the histogram above it, in the facets beside it, in a CSV export, in a saved view, in a log monitor and in live tail. Spans and errors use the same rules with their own reserved fields.
service:checkout status:errortimeout AND NOT service:billing@http.status_code:>=500@duration:[1s TO 5s]We chose this for two reasons. The first is people: a query you learn for searching can be reused unchanged as the condition of an alert, so there is no second language and no translation step where the meaning could drift. The second is correctness. A query that cannot be parsed is rejected with the position of the problem, for example Missing term at position 9, and values are bound as query parameters, so what you type reaches the database exactly as typed.
The live tail shows why a single definition matters. Tail filters rows in memory as they arrive, while search runs inside ClickHouse. Both use the same parsed query and the same semantics, and a test checks that the stored search and the tail agree, so the tail shows what a search over the same range would show.
The syntax has deliberate limits, 2,000 characters, 64 terms and 16 levels of nesting, which keep it predictable when many monitors run it every minute.
What is still plannedLink to section: What is still planned
Exponential histograms and metric summaries are not supported yet; gauges, sums and histograms are. Continuous profiling of your own services is planned for the product, as are monitoring integrations for your own databases and stack. Pulse’s data also has to be separated per organization before launch. You can follow these on the roadmap.
If you want to try the intake path, the OTLP page in the docs lists the endpoints, the headers and the standard SDK settings.