When a Kafka cluster is down, its admin client can fill your log

A Kafka AdminClient kept open while its cluster is down — the kind a monitoring tool, a console or a health check holds — logs Rebootstrapping with Cluster(…) at INFO thousands of times a second: about 90 MB a minute with the client's default settings once it settles, for as long as the cluster stays down. Since Kafka 4.0 this is the default behaviour. On that client, metadata.recovery.strategy=none stops it; what it gives up is below.

How it shows up

What you see	What it is
[AdminClient clientId=…] Rebootstrapping with Cluster(id = null, nodes = [host:9092 …], …) at INFO, over and over	The admin client going back to bootstrap.servers, once per attempt
A log that grows by gigabytes an hour while one cluster is unreachable	About 90 MB a minute per admin client with the defaults
Other work in the same process slowing down	With a synchronous log appender, once its output cannot keep up, every thread that logs waits for it
Timed out 296 remaining operation(s) during close, although you asked for one thing	Metadata calls the client queued for itself

What we measured

One admin client per run, pointed at a port nothing listens on, asked describeCluster once and then left open: two minutes in all, logging at INFO. The client jars came from Maven Central and were checked against its checksums.

Client	metadata.recovery.strategy	Log in two minutes	Rebootstrapping lines
3.9.2	none (its default)	45 KB	0
3.9.2	rebootstrap	135 MB	557,683
4.0.2, 4.1.2, 4.2.2, 4.3.1	rebootstrap (their default)	147 MB	567,873–571,187
4.3.1	none	46 KB	0

The loop starts at about a hundred lines a second and settles near 6,000 a second within a minute. It does not need a caller: a client that is opened and never asked anything loops as well. A long-lived producer pointed at the same closed port did not spiral — fewer than ten lines a second, steadily.

Why it happens

Rebootstrap arrived in Kafka 3.8 (KIP-899), switched off by default: when none of the brokers a client knows is available, it goes back to bootstrap.servers and starts over. Kafka 4.0 turned it on for every client (KIP-1102).

In the admin client, the thread that sends a metadata request first asks for a node to send it to. If no node is ready — and when the cluster is down, the only node it knows is the bootstrap address, waiting out its reconnect backoff — it rebootstraps, which logs the line above at INFO, and asks again. Nothing limits how often. Meanwhile the client keeps its own metadata calls pending and retries them; each retry goes through the same path. That is why, in our runs, the pace followed retry.backoff.ms and not the reconnect backoff.

What does not help

• Longer reconnect backoffs. The setting's own documentation suggests raising reconnect.backoff.ms and reconnect.backoff.max.ms and lowering socket.connection.setup.timeout.ms. With 5 seconds rising to 30, and a 3-second connection setup, the client logged 142 MB in two minutes — the same as without.
• Turning the logger down. With org.apache.kafka.clients.admin.internals.AdminMetadataManager at WARN the log stays quiet, but the loop runs on: 299 calls pending at close, and the admin thread spent 1.1 seconds of CPU in two minutes against 0.3 with none. A fair stopgap for a tool whose client settings you cannot change; not a fix.
• A longer retry backoff helps without stopping it: retry.backoff.ms=500 cut the two minutes to 8.4 MB.

The fix, and what it trades

On a long-lived admin client:

metadata.recovery.strategy=none

The client then logs one connection warning per reconnect attempt and waits for the brokers it knows. We started the cluster during a run, stopped it and started it again: the client was answering again within about seven seconds each time.

What you give up is what rebootstrap is for: a cluster whose every broker comes back at once under names the client has never seen. With none the client keeps trying the brokers it knew until you open a new client. Kafka's own description says rebootstrapping is useful when a client talks to the brokers "so infrequently that the set of brokers may change entirely before the client refreshes metadata" — a client that asks every minute is not that client. If yours is, keep rebootstrap, raise retry.backoff.ms, and close the client after its calls have failed for a while so the next use opens a new one.

Check your own

With your client version, your logging at INFO and a port nothing listens on:

import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeUnit;
import org.apache.kafka.clients.admin.Admin;

public class Check {
    public static void main(String[] args) throws Exception {
        Map<String, Object> props = new HashMap<>();
        props.put("bootstrap.servers", "localhost:19392"); // a port nothing listens on
        // props.put("metadata.recovery.strategy", "none"); // the fix: run again with this line in
        try (Admin admin = Admin.create(props)) {
            try {
                admin.describeCluster().clusterId().get(70, TimeUnit.SECONDS);
            } catch (Exception expected) {
                // the cluster is down: the call times out after a minute
            }
            Thread.sleep(60_000); // the client stays open, as a pooled one would
        }
    }
}

Count the Rebootstrapping lines, then uncomment the second put and run it again.

What BROKA does

BROKA keeps one admin client per registered Kafka cluster and asks it every minute, so a cluster that was down showed it to us first: 297 MB of log in two hours on a development stack, after which the service stalled waiting on its own log output. From release 1.2.0, BROKA builds its Kafka admin clients with metadata.recovery.strategy=none. The case it gives up is stated in the release notes: if every broker of a cluster is replaced at once under other names, BROKA finds the new ones when the connection's settings change or the service restarts.

What this article does not cover

• Consumers, and clients in other languages. We measured the admin client and a producer, not a consumer; clients built on librdkafka have their own rebootstrap.
• metadata.recovery.rebootstrap.trigger.ms, KIP-1102's other setting, which rebootstraps a client that has had no metadata for five minutes. The loop above takes the other path — no broker ready — and does not wait for it.

The Kafka page describes what BROKA does with a Kafka cluster.

Applies to Kafka's Java admin client 3.8 and later with rebootstrap, and 4.0 and later by default · in BROKA, from release 1.2.0 · Community and Commercial.