Filtering messages

Filters on a Kafka topic's Messages tab narrow a read on the broker as it reads: by text, pattern, JSONPath, script, or conditions on key, value and headers.

They sit in the toolbar above the records, beside the start point and the stop condition described in Reading messages, and they apply to the read you start with Run.

Two controls, one read

The toolbar holds two filters, and a read can use both.

• The filter box, with its mode chosen in the select beside it: Contains, Regex, JSONPath or Script. It asks one question of the record's decoded value. The key and the headers are not part of it, which is why its placeholder reads Value contains….
• Conditions, a button that opens a builder of typed conditions over the value, the key or a named header. The button shows how many conditions are set.

A record is returned only when it passes both. The cheaper tests run first — the box's text, pattern or path, then the conditions, themselves ordered from a header lookup to a text match to a pattern to a path, and a script last — so an expensive test runs only on a record nothing cheaper has already turned away. The order changes what a read costs, never what it returns.

The value tested is the decoded one. A record produced through Schema Registry is filtered as the JSON it decodes to, so an Avro topic can be searched for an order id (Schema-encoded records). A record with no value — a tombstone — never matches the filter box, because there is nothing to test; a does not exist condition on the value is how to find one.

Contains

Contains looks for your text anywhere in the decoded value and ignores case. It is the default, and the cheapest of the four.

Regex

Regex searches for a pattern anywhere in the value: the pattern has to describe part of the record, not all of it. Like Contains it ignores case; start the pattern with (?-i) to make it case-sensitive.

The broker's engine is the one that decides. Your browser also runs the pattern over the records already on screen, with its own engine, so the table narrows as you type. A pattern the browser cannot read — a possessive a++, say — is not an error: the ! beside the box says the table will not narrow as you type, and Run still sends it to the broker.

Each pattern works within a budget per record that grows with the record's size. A pattern that runs past it — nested quantifiers such as (a+)+ are the usual cause — abandons the read, with a message saying the regular expression took too long on a single record and was stopped, so the scan was abandoned rather than report records it never finished checking. Skipping that record instead would have put it among the ones you were told do not match. Simplify or anchor the pattern, then run again.

A refusal never repeats the filter you typed — it is in the box — because a filter is often the very values being searched for, and a refusal passes through the services' logs. A pattern that will not compile is refused with what is wrong and, where the engine says, the position.

JSONPath

JSONPath asks something of the document rather than of its letters. A record matches when the path selects anything: $.orderId finds every record that has an order id, and $[?(@.status=='FAILED')] finds the failed orders without also finding every record that merely mentions the word.

A value that is not a JSON object or array never matches, and nor does one that is not valid JSON: the path asked about a field, and a document that cannot be read has none. A bare field name typed where a path belongs, such as orderId, is read as $.orderId.

With JSONPath chosen, the note beside the box lists what it accepts. Filter expressions take the operators == != < <= > >= in nin contains subsetof anyof noneof size empty, and paths take the functions length() min() max() avg() sum() stddev() keys() first() last().

JSONPath's own regex operator, =~, is refused before anything is read: JSONPath's =~ operator is not supported in a filter, because its match runs with no time limit. Use a condition with the Regex operator on that path instead. The same refusal applies to a path inside a condition. A =~ inside a quoted value is text, and is not refused.

A path that will not compile is refused with the reason — or, where the parser's reason would quote the path back, only the position it stopped at — and with the shapes a path takes: $.orderId, $.items[0].sku, $[?(@.status=='FAILED')].

Script

Script is a JavaScript predicate, for the one thing the other filters cannot do: relate one field to another, as in value.items.length > 1 && !value.paid. Choosing it turns the box into a Script button, and the editor it opens takes a function body, run on the broker once per record. The body is handed:

Name	What it holds
key, value	The decoded key and value — parsed when the text is a JSON object or array, the text itself when it is not, null when the record has none
headers	The record's headers as a plain object, so headers["trace-id"] and "trace-id" in headers both work
offset, partition	Where the record is
timestamp	The record's timestamp, in milliseconds since the epoch
keySize, valueSize	The serialized sizes in bytes, as the broker reports them; -1 for an absent key or value

It must return something, and the result is read for truthiness: return value.items.length is accepted as written. The editor says what the engine takes: ES6 and later syntax — arrow functions, template literals, spread, ?., ??, ** and BigInt — with at, flat, includes, replaceAll and Object.entries, but not class.

A script runs in a sandbox, in a process of its own: whatever it does to that process, the service serving everyone else carries on. It cannot reach Java, files or the network, and one record's run cannot redefine the built-in objects the next record sees. At most four reads with a script run at once, and none starts while the service is short of memory; such a read is refused and can be tried again in a moment. Each record gets an instruction budget and a limit on recursion depth, and a script may be up to 8,000 characters. A script that will not compile is refused before anything is read, with the parser's reason. After that, three things abandon the read rather than skip a record:

• a script that runs past its budget, which is named with the partition and offset it stopped on;
• a script that needs more than 64 MB of memory, or runs for more than five seconds, on one record — named the same way;
• a script that throws — typically by reading a field of a record that lacks it — named with the record and the error. Guard an optional field: value && value.items.

A record the script could not judge is not counted as a non-match, because that would put it among the records you were told do not match.

Conditions

Conditions asks about a chosen part of the record rather than searching all of it — $.status equals FAILED, a header that exists, a retry count above three. Each row has four parts:

1. Scope — Value, Key or Header.
2. For Value and Key, an optional path into the decoded text ($.field); without one, the operator applies to the whole text. For Header, the header's name.
3. Operator — equals, does not equal, contains, matches regex, is greater than, is at least, is less than, is at most, exists, does not exist.
4. The value to compare against, disabled for exists and does not exist, which compare nothing.

Every text comparison ignores case, equals included. matches regex searches anywhere in the text, within the same per-record budget as the Regex box, and (?-i) makes it case-sensitive too.

The four numeric operators need a number to compare against, and a condition whose value is not one is refused. A refused condition — that one, or a path or pattern that will not compile — is named by its number in the list. A record whose field is not a number simply does not match: one odd record does not fail the read.

exists holds when the scope, or the path within it, has a value. does not exist is the one operator that holds for a record without it — "which records carry no trace id" is not a question a text search can ask.

A path that selects several values, such as $.items[*].sku, holds when any one of them satisfies the operator. A path over text that is not JSON selects nothing. The path box suggests the fields and header names the records already on screen carry, shallowest first, and still takes one they do not, because the range you have read is not the whole topic.

Conditions combine with and, and only and: a and b or c reads two ways, and a filter you cannot read is one you cannot trust. An either-or on one field is a single matches regex with an alternation, such as FAILED|REJECTED; a JSONPath filter expression can hold its own logic.

A half-filled row blocks the read rather than being dropped from it — Condition 2: name the header. or Condition 2: enter something to compare against. — because running only the complete rows would return more than the screen describes.

A condition on the key or on a header makes the broker decode the key, or copy the headers, of every record it reads rather than only of those it returns, so it is a heavier read on a large topic. A script does the same, since it is handed all of them.

Where the work happens

Every filter goes to the broker when you start a read, so a scan from the beginning finds matches anywhere in the log without sending every record to your browser. Contains and Regex also re-run in the browser over the records already on screen, so refining is instant. A JSONPath, a script and the conditions are evaluated by the broker alone — the browser has no sandbox for a script — so the table narrows when you press Run; with a JSONPath or a script chosen, the note beside the box says so.

The broker keeps the filter a read started with. Changing it during a read — or while following, when the toolbar's button reads Pause rather than Run — narrows what is on screen at once for Contains and Regex; the next Run searches the topic with the new one.

When the text you typed hides every record already received, the table says Nothing matches the filter, how many records arrived, and that you can clear the filter or press Run to search the topic for it.

Filters, limits and stopping

A non-matching record does not count against your limit. Read from the beginning with a limit of 200 and the read stops at the 200th match, however many records it passed over to find them.

Most recent is the exception. Its window is the newest records, as many as the limit, and a filter narrows that window rather than reaching past it. When a filtered Most recent read finds matches, BROKA says so: "Most recent" only looks back 200 records — switch to "From the beginning" to search the whole topic. When a filtered read finds none, the empty table names what was searched and offers the same switch.

A filter that matches nothing cannot make a read run forever. A bounded read stops after scanning 1,000,000 records — Stopped at the 1,000,000-record scan ceiling — narrow the range or the filter — or after 60 seconds — Ran out of time before reaching the end. Neither says it reached the end of the topic, because it did not; raising the limit would not get further, and narrowing the partitions, the start point or the filter does.

With an end time, a record past it is counted as scanned but is neither tested nor returned. While following, the limit does not stop the read: matches keep arriving until you stop it.

Masked fields

Commercial only. On a topic a data masking rule covers, a filter cannot be used to find out what is masked. For someone who may not read the topic unmasked, a JSONPath filter or a condition whose path reaches a masked field, and a condition on a masked key or a masked header, are refused before anything is read, naming the rule: The filter reads …, which the masking rule "…" covers, so it would reveal what is masked. Filter on a field you can see, or search the text, which runs on what you can see.

Every other filter runs on the records as you see them, masked: Contains, Regex, a Script, and a condition on the whole value, without a path. Each can match only what is visible, so searching for a value a rule hides finds nothing. Someone allowed to read the topic unmasked filters it as before.

What the audit trail keeps

Commercial only. A Community installation records every change and every refusal, and no reads.

A read's audit record says whether a filter was set, which mode it used and how many conditions it had — never the expression or the values compared against, which are often the very identifiers the record is sensitive for. The rest of what a read records is under Reading is recorded.

Permissions

Filtering needs no permission of its own: whoever can read a topic can filter it (Permissions).

Commercial only: a topic a data masking rule covers adds one. Filtering on a masked field takes Read masked contents unmasked (messages.read.unmasked) for the topic — on its environment or connection, or by a topic grant whose pattern matches it.