Cloudflare said on Monday that it had rebuilt the system that turns customers’ DNS changes into data for its servers, cutting the typical wait for an update roughly in half and reducing peak memory use by 85 percent.
The work centers on the Zone Builder, the service that takes in about 700 DNS record changes a second, up from about 250 in 2022, plus about 100 other events such as changes to zone settings. Some customer zones now hold more than 20 million records, and the company expects them to grow toward hundreds of millions, according to a post on Cloudflare’s engineering blog. The post isn’t signed.
Volume wasn’t the problem. Settings were. In 2022 Cloudflare taught the Zone Builder to rebuild only the records a change touched, and those builds ran 150 times faster than full rebuilds on average and 4,250 times faster on a test zone with one million records, the company reported at the time. A setting change, like turning Universal SSL on or off, still set off a full rebuild. That meant reading, encoding and comparing every record in the zone while holding all of it in memory. The largest zones don’t fit.
The company’s answer is a planner that picks the smallest build it can prove is safe, and one scheduler that merges queued work for the same zone. The planner chooses among five build types, from rebuilding a few DNS names to purging a zone or rebuilding all of it. The test is strict: every key a targeted build writes has to match, byte for byte, what a full build would have written. “If we cannot prove that, we do a full build,” the post said.
Over a 24-hour window, 86.81 percent of builds touched only specific names, and 92.95 percent avoided a full rebuild. Full builds made up 7.04 percent of the total, down from 11.88 percent, a cut of about 40 percent. The ones left average 0.35 seconds.
Average memory use fell from 49 GiB to 7 GiB, and the peak from 79 GiB to 12 GiB. With that room, engineers shortened delays the pipeline had always kept for resilience. The median time from a DNS change to its data landing in Quicksilver, the company’s distributed key-value store, went from about 7.7 seconds to about 6.4 after one change to how events are collected, then to about 3.3 seconds after shorter waits between rebuilds. The largest spike in 99th-percentile delay fell from 14.6 minutes to 1.21 minutes.
Kafka, the messaging system that feeds the pipeline, used to be consumed at anywhere from about 300 to 1,550 messages a second while new messages arrived at about 600 to 800. Consumption now keeps pace.
There is a tradeoff for the biggest customers. Above a fixed record count, the Zone Builder never falls back to a full rebuild. It rebuilds the bare domain, the shared metadata and every name the queued events identify. A full build of those zones can’t finish, the post said, and while it retries, it holds up every other change on its Kafka partition. In rare cases, such as enabling a zone on another network, engineers can rebuild individual names by hand.
Part of the difficulty is that one domain can briefly map to several internal Cloudflare zones during onboarding, offboarding and other changes, and they all share one metadata key. The builder also reads from database replicas that can lag behind the events, so it checks fresh state just before running and rejects reads from replicas that haven’t caught up.
Before the rollout, a mirror Zone Builder ran on live data but wrote to a separate store, and a comparer sampled 20 percent of zones. It showed a 0.31 percent difference rate across 234,313 matched zones. Cloudflare then switched over free zones first and paid zones after.
Full builds aren’t going away. A zone published for the first time still needs one, as do changes that can affect records anywhere in a zone and events that update only a timestamp without saying why. “The planner never turns uncertainty into less work,” the post said. The company said it would move more event types to targeted builds as it learns what each one changes.