More than 110 million log records a second leave Cloudflare’s network for customer SIEMs, analytics platforms and object storage through its Logpush service, and each of them can now pass through a customer-written SQL query first. The feature, called Transformers, went generally available during the company’s recent Birthday Week, alongside Logpush itself becoming available on all self-serve plans; a post on Cloudflare’s engineering blog now explains how the machinery works underneath.
Filter it before you pay to store it
The problem Transformers addresses is mundane and expensive. Observability teams routinely strip health checks, omit sensitive fields and reshape records for dashboards, and until now that meant a separate processing service to deploy, scale, monitor and maintain between Cloudflare and the destination. Teams also pay to export, ingest and store records they discard the moment they arrive, so dropping them before delivery cuts downstream volume and cost directly. A team collecting logs from several CDNs can add a static provider field to normalise records before they hit an analytics platform.
A Transformer is a SQL query scoped to one Logpush dataset. Records that fail the WHERE clause are removed; for the rest, the SELECT output becomes the exported record. Input and output are both NDJSON — this reshapes records, it does not convert serialization formats. Customers build and preview queries in a dashboard tool called Transformer Studio or via the API, attach them to a compatible NDJSON Logpush job on the same dataset, and every saved update creates a new version for review.
The post’s example takes an HTTP request log and rewrites it to a cross-CDN schema: the timestamp becomes epoch seconds; the client IP 203.0.113.42 is masked by regexp_replace to 203.0.XXX.XXX; the request method is uppercased; a literal ‘cloudflare’ provider field is added; host and path are nested into a named struct; a boolean flags status codes of 500 or above; and the WHERE clause drops the internal and health-check hostnames plus any path under /healthz or /metrics.
The pipeline, as described: Buftee (Cloudflare’s distributed log-based message broker) | compressed Cap’n Proto batches v Bufstream pod, written in Go: decompress, parse, build a typed Apache Arrow RecordBatch | Arrow IPC buffer; Go lends Rust a pointer for the duration of one cgo call v Apache DataFusion, written in Rust: plan and execute the customer’s SQL, vectorized and columnar | Rust-owned output buffer, copied into Go memory, then the Rust allocation is explicitly freed v NDJSON batch delivered to the job’s destination
The prototype lost to a network hop
Cloudflare’s first attempt ran customer-defined WebAssembly transforms inside Workers, its globally distributed isolated compute platform. The problem is structural: invoking a Worker inserts a remote call into the export hot path, and that dependency means transient timeouts and retries, which cut delivery throughput and add latency. For a service whose requirements are reliable delivery, high throughput, predictable resource use and no new failure modes, in-process execution won. Bufstream, the delivery service, runs across hundreds of Kubernetes pods, each claiming a time slice of data from Buftee. Job volume follows a Pareto-like distribution — most jobs are small, while a small number push millions of logs per second — so Bufstream uses per-job transform parallelism to let high-volume jobs execute batches concurrently.
Go to Rust, one columnar batch at a time
For the in-process engine Cloudflare evaluated several SQL options and picked Apache DataFusion — a Rust query engine with a SQL planner and vectorized, columnar execution built on Apache Arrow — which already powers Cloudflare Basin. Bufstream is Go, so batches cross a foreign-function boundary exposed as a C ABI and called through cgo. Arrow IPC, the columnar format’s interchange protocol, is what makes that crossing cheap: Rust reads typed RecordBatch values with an Arrow StreamReader rather than parsing rows out of JSON and rebuilding columns, and field types, nullability and nested values survive the boundary intact, so schema validation against the dataset definition still holds. Memory ownership across the two runtimes is made explicit: Go’s input pointer is valid only for the call, Rust never retains it, and the transformed output is copied into Go-managed memory before the Rust allocation is released. Crossing still costs serialization and decoding, but the work is batch-oriented rather than per-record.
On performance, Cloudflare reports one benchmark: a prepared Transformer over a synthetic 5,000-row http_requests batch, executed from a single Go goroutine, took about 3 ms — roughly 1.6 million rows per second. That is the vendor’s own figure, on synthetic data, measuring the SQL execution step in isolation; decompression, the Arrow IPC crossing and destination delivery all sit outside it, and production jobs running concurrent batches will contend for CPU in ways a single-goroutine test does not show. The number establishes that the engine is not obviously the bottleneck; it says little about end-to-end overhead per job.
Cloudflare has been squeezing this pipeline for years — it previously described trimming 36 GB/s of traffic through schema, compression and sampling work while event volume kept climbing.
What the post does not quantify is the customer’s side of the ledger: how much CPU a Transformer adds per job at millions of records per second, what limits apply to query complexity, and whether filtering before export shows up as a measurable reduction in SIEM and storage bills. Those are the numbers that decide whether the 3 ms benchmark matters.