Apr 30, 2026 · @raxyte · #5766
The from_gcs operator has been renamed to from_google_cloud_storage so
that its name matches the new to_google_cloud_storage writer:
// Before:
from_gcs "gs://my-bucket/data/**.json"
// After:
from_google_cloud_storage "gs://my-bucket/data/**.json"
Update test suites that reference from_gcs in requires.operators
accordingly.
The default message expression of the to_kafka operator is now
this.print_ndjson() instead of this.print_json(). Kafka messages are
single-line records by default, so each event is now emitted as a single
NDJSON line:
{"timestamp":"2024-03-15T10:30:00.000000","source_ip":"192.168.1.100","alert_type":"brute_force"}
instead of pretty-printed multi-line JSON.
To restore the previous behavior, pass message=this.print_json()
explicitly.
The yara operator no longer accepts the blockwise argument. Instead, it
buffers the entire input as one contiguous byte sequence and runs the YARA
scan when the input ends. Matches can therefore span chunk boundaries, but
yara is now only suitable for finite byte streams. Don't use it on
never-ending inputs.
The rule argument now also accepts a single string in addition to a list
of strings:
from_file "evil.exe", mmap=true {
yara "rule.yara"
}
Removed:
yara ["rule.yara"], blockwise=true
Apr 30, 2026 · @aljazerzen · #5880
The measure operator no longer accepts the real_time argument. The
operator's emission cadence is now governed entirely by the executor's
backpressure, so the option no longer has a meaningful effect.
Remove real_time=true or real_time=false from your pipelines:
// Before:
measure real_time=true
// After:
measure
Apr 30, 2026 · @raxyte · #6001
The filesystem and cloud object reader operators (from_file, from_s3,
from_azure_blob_storage, from_google_cloud_storage) no longer accept the
path_field option. Instead, the parsing subpipeline now has access to a
$file let-binding describing the source file:
| Field | Type | Description |
| :------ | :------- | :--------------------------------------- |
| path | string | The absolute path of the file being read |
| mtime | time | The last modification time of the file |
To attach the source path to each event:
// Before:
from_file "/data/*.json", path_field=source
// After:
from_file "/data/*.json" {
read_json
source = $file.path
}
This makes per-file metadata available throughout the parsing subpipeline rather than only on emitted events.
Apr 30, 2026 · @aljazerzen · #5953
The from_http operator no longer doubles as an HTTP server. It is now a
pure HTTP client that issues one request and returns the response.
For accepting incoming HTTP requests, use the dedicated accept_http
operator instead:
// Before:
from_http "0.0.0.0:8080", server=true { read_json }
// After:
accept_http "0.0.0.0:8080" { read_json }
In the parsing subpipeline, the response metadata is now exposed as the
$response let-binding instead of being written into a metadata_field:
from_http "https://api.example.com/status" {
read_json
status_code = $response.code
server = $response.headers.Server
}
Additionally, the url and headers arguments are now resolved as
secrets, so you can pass
secret names instead of hardcoding tokens or sensitive URLs.
Apr 30, 2026 · @aljazerzen · #6066
The new accept_opensearch operator starts an OpenSearch-compatible HTTP
server and turns incoming Bulk API requests into events:
accept_opensearch "0.0.0.0:9200"
publish "events"
The operator buffers each bulk request body up to max_request_size,
optionally decompresses it based on the Content-Encoding header, parses the
NDJSON payload, and emits the resulting records. Set keep_actions=true to
also keep the OpenSearch action objects (e.g., {"create": ...}) in the
stream.
The from_opensearch operator has been removed. Use accept_opensearch
instead. The elasticsearch:// and opensearch:// URL schemes now dispatch
to accept_opensearch via from.
Two new operators provide first-class FTP and FTPS support with parsing and printing subpipelines:
from_ftp downloads bytes from an FTP or FTPS server and forwards them to
the parsing subpipeline.to_ftp uploads bytes produced by the printing subpipeline to an FTP or
FTPS server.from_ftp "ftp://user:pass@ftp.example.org/path/to/file.ndjson" {
read_ndjson
}
to_ftp "ftp://user:pass@ftp.example.org/a/b/c/events.ndjson" {
write_ndjson
}
The load_ftp and save_ftp operators have been removed, and the ftp://
and ftps:// URL schemes no longer dispatch via from and to. Use
from_ftp and to_ftp directly.
The neo to_splunk implementation now accepts queue="indexing" and
queue="typing" for selecting the Splunk HEC processing queue. The default
indexing path keeps Splunk's regular HEC behavior, while typing sends the
Splunk parsingQueue hint in HEC event envelopes for receivers that support
this non-standard HEC metadata.
The default is queue="indexing". The typing queue is rejected with
raw=..., because Splunk's raw HEC endpoint sends raw requests to the indexer
queue.
May 18, 2026 · @jachris, @claude
The read_csv, read_tsv, read_ssv, and read_xsv operators now accept an
auto_fill=true option. When set, the parser silently fills missing trailing
columns with null instead of emitting a warning, which is useful when working
with feeds that legitimately omit optional trailing fields.
The from_http operator now supports returning request records from
paginate lambdas. This lets APIs keep pagination state in the next request
body or headers instead of only in the next URL:
from_http "https://opensearch.example.com/logs/_search",
method="post",
body={size: 500, query: {match_all: {}}},
paginate=(x => {
body: {
size: 500,
query: {match_all: {}},
search_after: x.hits.hits[-1].sort,
},
} if x.hits.hits != []) {
read_json
}
Returned request records can patch url, method, headers, and body.
Missing fields inherit from the current request, and body: null clears the
body.
May 16, 2026 · @mavam, @codex · #6190
The metrics operator now accepts shape="prometheus" to emit metrics from
metrics plugins as canonical {metric, value, timestamp, labels, type, unit}
records. The default remains shape="raw", which preserves the existing
tenzir.metrics.* schemas.
The new write_all operator concatenates one selected string or blob field
into raw bytes:
from_file "/tmp/report.pdf" {
read_all binary=true
}
to_file "/tmp/report-copy.pdf" {
write_all data
}
Use it to copy binary payloads, reconstruct byte streams after event processing, or write string fields without separators or escaping.
May 15, 2026 · @mavam, @codex · #6181
The new repeat function repeats a string a given number of times:
message = "na".repeat(8)
{
message: "nananananananana",
}
The from_http operator now accepts requests without an explicit parser
subpipeline when Tenzir can infer the response format from the Content-Type
header or URL extension:
from_http "https://example.com/events.json"
Explicit parser subpipelines continue to take precedence over inferred formats.
May 14, 2026 · @mavam, @codex · #6180
Tenzir now supports reading from and writing to CloudWatch Logs with the new
from_amazon_cloudwatch and to_amazon_cloudwatch operators. The source can
subscribe to live streams with mode="live", search historical log groups with
mode="search", or replay one stream with mode="replay".
from_amazon_cloudwatch "/aws/lambda/api", mode="search", filter="ERROR"
The default sink can send events with PutLogEvents, including configurable
batching, timestamp handling, parallel requests, and AWS IAM authentication via
aws_iam. The sink can also write to the CloudWatch HTTP ingestion endpoints by
setting method to json, ndjson, or hlc, with either SigV4 or bearer-token
authentication.
to_amazon_cloudwatch "/tenzir/events",
stream="default",
payload=message,
timestamp=ts
May 13, 2026 · @mavam, @codex · #6167, #6174
The from_sqs operator now gives you explicit control over how messages are
received from SQS. Use keep_messages=true to inspect or replay messages
without removing them from the queue, batch_size=<1..10> to control how many
messages each receive request may return, and visibility_timeout=<duration> to
override the queue visibility timeout for received messages:
from_sqs "events", keep_messages=true, batch_size=10, visibility_timeout=30s
By default, from_sqs keeps deleting each received message after emitting it.
With keep_messages=true, SQS makes the message visible again after the queue's
visibility timeout.
May 12, 2026 · @mavam, @codex · #6165, #6179, #6182
Tenzir now includes a Microsoft Graph source operator for reads from Microsoft Graph v1.0 and beta collections with app-only Microsoft Entra authentication and OData pagination.
For example, you can read Entra ID sign-in logs with client credentials and push down OData query options:
from_microsoft_graph "auditLogs/signIns",
auth={
tenant_id: "contoso.onmicrosoft.com",
client_id: "00000000-0000-0000-0000-000000000000",
client_secret: secret("ms-graph-client-secret"),
},
odata={
filter: "createdDateTime ge 2026-04-24T00:00:00Z",
select: ["id", "createdDateTime", "userPrincipalName", "status"],
top: 1000,
}
The operator emits each object from the response value array as a separate event and follows @odata.nextLink until the collection is exhausted.
The operator can also use Microsoft Graph delta queries with delta=true, storing the returned @odata.deltaLink in memory and polling it with a configurable poll_interval. OData query options apply to the initial delta request only, subject to Microsoft Graph's resource-specific support, and subsequent polls use the opaque delta link exactly as Microsoft Graph returned it.
It also retries throttled and transient Microsoft Graph requests, respecting Retry-After when present.
Pipeline metrics now report event throughput alongside byte throughput for pipelines running on the new executor:
metrics "pipeline"
summarize ingress_events=sum(ingress.events), ingress_bytes=sum(ingress.bytes), egress_events=sum(egress.events), pipeline_id
sort -egress_events
This makes node metrics distinguish the amount of data transferred from the number of events processed.
May 8, 2026 · @IyeOnline, @codex
The new internal_memory_size function estimates the size of each event in
bytes:
size = internal_memory_size(this)
This is useful for building pipelines that inspect or route events based on their approximate in-memory payload size.
May 8, 2026 · @mavam, @codex · #6139
from_amqp now accepts a queue_arguments record for RabbitMQ queue declaration arguments:
from_amqp "amqp://broker/vhost",
queue="events",
queue_arguments={
"x-queue-type": "quorum",
"x-quorum-initial-group-size": 3
}
Use this to declare queues with broker-specific settings such as quorum queues, maximum lengths, message TTLs, single active consumers, and dead-letter exchanges.
TQL now supports statement-level match blocks for branching on patterns:
match action {
"accept" | "allow" => { verdict = "allowed" }
"deny" | "drop" => { verdict = "blocked" }
_ => { verdict = "unknown" }
}
Patterns can be constants, exclusive ranges, alternatives separated by |, or
the final wildcard _. Every match must include an unguarded final wildcard
arm, so Tenzir can prove at compile time that all possible values are covered.
This provides a concise alternative to long else if chains when routing events
by field value.
The write_feather operator now supports compression_type="uncompressed"
to disable compression entirely. Previously, only zstd and lz4 were
accepted:
to_file "events.feather" {
write_feather compression_type="uncompressed"
}
The to_splunk operator gains three new options for richer HEC metadata.
Use time= to set the per-event Splunk timestamp from an expression that
evaluates to a Tenzir time or a non-negative epoch in seconds:
from {message: "login succeeded", observed_at: 2026-04-24T08:30:00Z}
to_splunk "https://localhost:8088",
hec_token=secret("splunk-hec-token"),
time=observed_at
Use fields= to attach indexed HEC fields. The expression must evaluate to
a flat record whose values are strings or lists of strings:
to_splunk "https://localhost:8088",
hec_token=secret("splunk-hec-token"),
event={message: message},
fields={user: user, tags: tags}
Use raw= to send already-formatted text to the HEC raw endpoint
(/services/collector/raw). The raw expression must evaluate to a
string. Multiple events in one request are separated by newlines, and
request-level metadata such as host, source, sourcetype, index, and
time is sent as query parameters:
to_splunk "https://localhost:8088",
hec_token=secret("splunk-hec-token"),
raw=line,
source=source,
sourcetype="linux_secure"
raw and event are mutually exclusive; fields is not supported with
raw.
The dns_lookup operator now caches DNS results and reuses them across
lookups. Forward-lookup results gain a ttl field that shows the remaining
lifetime of the cached answer:
from {host: "example.com"}
dns_lookup host
If Tenzir cannot initialize DNS resolution at all, the operator now emits an
error and stops instead of writing null results for every event.
Individual failed or timed-out lookups still produce null, as before.
Apr 30, 2026 · @jachris · #5821
The parallel operator gains two enhancements:
The jobs argument is now optional and defaults to the number of available
CPU cores:
subscribe "events"
parallel {
parsed = data.parse_json()
}
The new route_by argument routes events to workers deterministically by
key. Events with the same route_by value always go to the same worker,
which is required for stateful subpipelines like deduplicate or
summarize:
subscribe "events"
parallel route_by=src_ip {
deduplicate src_ip, dst_ip, dst_port
}
Additionally, parallel may now be used as a source operator (without
upstream input). This spawns multiple independent instances of the
subpipeline, which is useful for running the same source pipeline with
concurrent connections.
Apr 30, 2026 · @raxyte · #6036
The from_file operator now accepts an mmap=bool option that uses
memory-mapped I/O for reading local files instead of regular reads. This can
improve performance for large files:
from_file "/var/log/large.json", mmap=true {
read_json
}
Defaults to false.
The new read_tql operator parses an incoming byte stream of TQL-formatted
records into events. Each top-level record expression becomes one event:
load_file "events.tql"
read_tql
The input format matches the output of write_tql, so read_tql is useful
for round-tripping data through TQL notation, reading TQL-formatted files,
or processing data piped from other Tenzir pipelines.
Apr 30, 2026 · @raxyte · #6053
Four new high-level writer operators serialize events to local filesystems and cloud object stores with rotation, hive-style partitioning, and per-partition unique filenames:
to_file writes to a local filesystem.to_s3 writes to Amazon S3.to_azure_blob_storage writes to Azure Blob Storage.to_google_cloud_storage writes to Google Cloud Storage.Each takes a printing subpipeline, a URL with optional ** and {uuid}
placeholders, and rotation parameters. The ** placeholder expands into a
hive partitioning hierarchy based on partition_by, and {uuid} ensures
each partition gets unique destination names:
subscribe "events"
to_s3 "s3://my-bucket/year=**/month=**/{uuid}.json",
partition_by=[year, month] {
write_ndjson
}
Files rotate automatically when the configured max_size or timeout is
reached, so long-running pipelines do not produce single huge objects.
Apr 30, 2026 · @aljazerzen · #6019
The new to_http operator sends each input event as an HTTP request to a
webhook or API endpoint. By default, it JSON-encodes the entire event as the
request body and sends it as a POST:
subscribe "alerts"
to_http "https://example.com/webhook"
to_http shares its options with from_http and http: configure
method, body, encode, headers, TLS, retries, and pagination per
request. Use parallel to issue multiple concurrent requests when the target
endpoint can keep up with a single pipeline.
This is useful for pushing alerts to webhooks, forwarding events to SIEMs, and calling external APIs once per event.
Apr 30, 2026 · @aljazerzen · #6070
The new serve_http operator starts an HTTP server and broadcasts the bytes
produced by a nested pipeline to all connected clients:
from_file "example.yaml"
serve_http "0.0.0.0:8080" {
write_ndjson
}
Clients connect with a GET request and receive a continuous HTTP response
body. Pick the wire format with the nested pipeline: write_ndjson for
NDJSON streams, write_lines for plain text, and so on. The operator does
not buffer output for clients that connect later—each client receives the
bytes produced after it connects. TLS, connection limits, and graceful
disconnect are all configurable.
The new from_nic operator captures packets from a network interface and
emits them as events directly:
from_nic "eth0"
Without an explicit subpipeline, from_nic parses the captured PCAP byte
stream with read_pcap. Provide a subpipeline when you want to change how
the byte stream is parsed. Use the filter option to apply a Berkeley Packet
Filter (BPF) expression so libpcap drops unwanted traffic before parsing:
from_nic "eth0", filter="tcp port 443"
The companion read_pcap and write_pcap operators have been refreshed:
read_pcap now also emits a pcap.file_header event when
emit_file_headers=true, which write_pcap consumes to preserve the
original timestamp precision and byte order. The pcap.packet schema's
time.timestamp field is now a top-level timestamp field, and data is
now a blob.
Apr 30, 2026 · @mavam · #5744, #6017
Tenzir now has dedicated TCP source and sink operators that match the same client/server split as the HTTP operators:
from_tcp connects to a remote TCP or TLS endpoint as a client.accept_tcp listens on a local endpoint and spawns a subpipeline per
accepted connection. Inside the subpipeline, $peer.ip and $peer.port
identify the connecting client; with resolve_hostnames=true,
$peer.hostname is also available from reverse DNS.to_tcp connects to a remote endpoint and writes serialized bytes.serve_tcp listens for incoming connections and broadcasts pipeline output
to all connected clients.Each operator takes a parsing or printing subpipeline so connection management, framing, and serialization stay separate concerns:
accept_tcp "0.0.0.0:8090" {
read_json
}
to_tcp "collector.example.com:5044" {
write_json
}
from_tcp and to_tcp reconnect with exponential backoff on connection
failure. All four operators support TLS via the tls option.
The legacy load_tcp and save_tcp operators are now deprecated. The
tcp:// and tcps:// URL schemes still dispatch to them via from and
to.
Apr 30, 2026 · @raxyte · #5731
The new from_stdin operator reads bytes from standard input through a
parsing subpipeline:
from_stdin {
read_json
}
This is useful when piping data into the tenzir executable as part of a
shell script or command chain.
Apr 30, 2026 · @IyeOnline
The new anonymize operator generates synthetic events that share the schemas
of its input. The operator first samples a configurable number of input events
to learn what schemas are present and to summarize their values, and then
replaces the input with generated events that match those schemas:
subscribe "events"
anonymize count=1000
By default, generated values follow the aggregate statistics of the sampled
input: null rates, list lengths, numeric ranges, time and duration ranges,
boolean and enum frequencies, string and blob lengths and byte frequencies, IP
address family frequencies, and subnet prefix length frequencies. Use
fully_random=true to ignore those statistics and instead pick values
uniformly from each type's full range. The optional seed argument makes
output reproducible.
Use anonymize to share representative event traces without leaking the
underlying values.
Apr 30, 2026 · @jachris · #5980
The new group operator routes events with the same key through a shared
subpipeline. Inside the subpipeline, $group refers to the key for that
subpipeline:
group tenant {
summarize count()
}
The subpipeline either emits events—which are forwarded as the operator's
output—or ends with a sink, in which case group itself becomes a sink. Use
group when you need keyed routing through a stateful subpipeline, such as a
per-tenant sink or a per-session transformation. For grouped aggregations,
keep using summarize.
Apr 30, 2026 · @jachris · #5981
The new each operator runs a fresh subpipeline for every input event. The
event is bound to $this inside the subpipeline so it can parametrize the
nested logic on a per-event basis:
from [
{file: "a.json"},
{file: "b.json"},
]
each {
from $this.file
}
The subpipeline takes no input from each. It either emits events—which are
forwarded as the operator's output—and may also end with a sink, in which case
each itself becomes a sink.
Use each for per-event jobs such as a lookup, an export, or a sink whose
source depends on the incoming event. For keyed streams that should keep one
subpipeline alive per key, use group instead.
The from_http operator now supports paginate="odata" for
OData collection
responses such as Microsoft Graph:
from_http "https://graph.microsoft.com/v1.0/users",
headers={"ConsistencyLevel": "eventual"},
paginate="odata" {
read_json
}
This mode emits the objects from the response body's top-level value array
and follows top-level @odata.nextLink URLs until no next link is present. The
next link can be absolute or relative to the current response URL.
Tenzir can now consume from and publish to NATS JetStream subjects with
from_nats and to_nats.
Use from_nats to receive one event per message. The raw payload appears in the
message blob field, and metadata_field attaches NATS metadata:
from_nats "alerts", metadata_field=nats
parsed = string(message).parse_json()
Use to_nats to publish one message per event. By default, the operator
serializes the whole event with this.print_ndjson():
from {severity: "high", alert_type: "suspicious-login"}
to_nats "alerts"
Both operators support configurable connection settings, authentication, and
the standard Tenzir tls record.
Feb 6, 2026 · @mavam, @claude · #5721, #5738
The from_mysql operator lets you read data directly from MySQL databases.
Read a table:
from_mysql table="users", host="localhost", port=3306, user="admin", password="secret", database="mydb"
List tables:
from_mysql show="tables", host="localhost", port=3306, user="admin", password="secret", database="mydb"
Show columns:
from_mysql table="users", show="columns", host="localhost", port=3306, user="admin", password="secret", database="mydb"
And ultimately execute a custom SQL query:
from_mysql sql="SELECT id, name FROM users WHERE active = 1",
host="localhost",
port=3306,
user="admin",
password="secret",
database="mydb"
The operator supports TLS/SSL connections for secure communication with MySQL
servers. Use tls=true for default TLS settings, or pass a record for
fine-grained control:
from_mysql table="users", host="db.example.com", database="prod", tls={
cacert: "/path/to/ca.pem",
certfile: "/path/to/client-cert.pem",
keyfile: "/path/to/client-key.pem",
}
The operator supports MySQL's caching_sha2_password authentication method and automatically maps MySQL data types to Tenzir types.
Use live=true to continuously stream new rows from a table. The operator
tracks progress using a watermark on an integer column, polling for rows above
the last-seen value:
from_mysql table="events", live=true, host="localhost", database="mydb"
By default, the tracking column is auto-detected from the table's auto-increment primary key. To specify one explicitly:
from_mysql table="events", live=true, tracking_column="event_id",
host="localhost", database="mydb"
The from_sqs and to_sqs operators now derive the AWS region from a queue
URL when aws_region is not set, so passing a full URL such as
https://sqs.us-west-2.amazonaws.com/123456789012/my-queue works without
having to specify the region again:
from_sqs "https://sqs.us-west-2.amazonaws.com/123456789012/my-queue"
Previously, this would fall back to the SDK default region and fail with a
SigV4 signature mismatch. Explicit aws_region, resolved IAM credentials, and
the SDK default still apply in that order when the URL has no region (for
example VPC endpoints, LocalStack, or an AWS_ENDPOINT_URL override).
SQS API errors and HTTP failures now also include the endpoint URL in their log lines and diagnostic notes, which makes it easier to tell which queue produced an error when multiple SQS pipelines run side by side.
May 12, 2026 · @mavam
The chart_bar and chart_pie operators now preserve the incoming row order
for categorical x-axis values such as strings, IP addresses, and subnets. This
allows users to control bar order with regular TQL operators such as sort
before charting.
Apr 30, 2026 · @aljazerzen · #5878, #5906
The batch operator now maintains separate buffers for each distinct
schema. Each buffer has independent timeout tracking and fills until reaching
the limit, at which point it flushes immediately. Previously, mixed-schema
streams could stall waiting for a single combined buffer to fill.
The timeout argument now defaults to 1min instead of an infinite
duration, so buffered events are flushed at least once per minute when no new
events arrive.
Apr 15, 2026 · @lava
We added a new operator to accept data from incoming HTTP connections.
The server option of the from_http operator is now deprecated.
Going forward, it should only be used for client-mode HTTP operations,
and the new accept_http operator should be used for server-mode
operations.
May 20, 2026 · @raxyte · #6210
The compaction plugin no longer fails to start with
module <package> not found when a rule's pipeline references an
operator defined by an installed package. Previously, depending on the
order in which the node's components were initialized, the compactor's
eager rule-pipeline parse could run before the package manager had
published its operator modules to the global registry.
May 12, 2026 · @lava
Time-based compaction rules no longer cause the node to reprocess data that has already been compacted in a previous run.
May 8, 2026 · @tobim, @codex · #6149
Packages can now include a top-level metadata field for data consumed by external tools. Unknown package keys still fail validation, and the error now points users to metadata for non-engine package data.
Apr 24, 2026 · @mavam, @codex · #6081
The to_sentinelone_data_lake operator now works in pipelines that run on the new executor. Previously, using it there failed before the pipeline could send events.
from {message: "hello"}
to_sentinelone_data_lake "https://example.com", token="TOKEN"
Mar 31, 2026 · @jachris, @mavam, @codex · #5963
The drop_null_fields operator is now much faster on heterogeneous input with many changing null patterns.
flow_id that mix signed and unsigned values no longer cause unnecessary table-slice splits. It also extends ocsf::derive to handle list-valued enum fields for full bidirectional OCSF enum normalization.
Apr 30, 2026 · @jachris, @mavam, @codex · #5354
ocsf::derive now derives OCSF enum sibling fields for lists, not just scalar enum fields. For example, DNS answers with flag_ids: [1, 3, 4] now also get flags: ["Authoritative Answer", "Recursion Desired", "Recursion Available"], and the reverse direction works for flags to flag_ids as well.
May 5, 2026 · @IyeOnline, @claude
Parsing data that mixes int64 and uint64 values in the same field no longer
produces unnecessary table-slice splits, improving batching performance. Fields
like flow_id that are always non-negative but occasionally exceed the signed
integer limit of 2^63 − 1 are now merged into a single uint64 column where
possible, instead of being emitted as separate slices.
May 6, 2026 · @mavam, @codex · #6128
Empty if branches no longer crash when running pipelines with the new executor. For example, if false {} now behaves like an empty pass-through branch instead of triggering an internal assertion failure.
Configured startup pipelines can now reference operators from static packages reliably. Previously, such pipelines could fail during node startup with module <package> not found, even though the same package operator worked when run manually after startup.
May 2, 2026 · @mavam, @codex · #6108
Package UDOs now load correctly when a typed string default looks like a TQL pattern, such as default: "/tmp-data/".
Previously, loading such a package could abort with an unexpected internal error before any pipeline ran.
]]>Apr 29, 2026 · @mavam, @codex · #6098
ClickHouse connection errors caused by TLS/plaintext mismatches now include the TLS notes and hint again. This helps identify when to_clickhouse is using TLS against a plaintext ClickHouse endpoint and suggests setting tls=false when appropriate.
Apr 28, 2026 · @tobim, @codex · #6086
Default retention policies now continue deleting metrics and diagnostics as their timestamps age into the retention window, even when older and newer events share a partition.
Previously, a partition that still contained newer events after retention could be skipped by later retention runs, leaving those events behind after they expired.
]]>Apr 27, 2026 · @tobim, @codex · #6082
Static musl builds of tenzir no longer crash on deeply nested generated TQL
expressions.
This affected generated pipelines with deeply nested expressions, for example rules or transformations that expand into long left-associated operator chains.
The tenzir binary now links with a larger default thread stack size on musl,
which brings its behavior in line with non-static builds for these pipelines.
The subnet function now accepts typed IP addresses, plain IP strings, and
existing subnet values with an optional prefix length:
from {source_ip: 10.10.1.124}
net = subnet(source_ip, 24)
This returns 10.10.1.0/24 without converting the IP address to a string first.
When you omit the prefix, IPv4 addresses become /32 host subnets and IPv6
addresses become /128 host subnets.
The unroll operator no longer crashes when expanding very large lists into output that exceeds Arrow's per-array capacity.
The files operator now skips unreadable child directories during recursive traversal, emits a warning for each skipped directory by default, and continues listing accessible siblings. Set skip_permission_denied=true to ignore permission-denied paths silently: this suppresses warnings for skipped child directories and still makes an unreadable initial directory produce no events instead of an error. Non-permission filesystem errors continue to fail the pipeline.
Apr 23, 2026 · @IyeOnline
We fixed an issue in the context::enrich operator that did cause unbounded
memory growth.
Apr 23, 2026 · @tobim, @codex · #6068
Tenzir no longer segfaults on some very deep left-associated boolean
expressions in where clauses due to source-location handling.
Both list and record indexing in TQL now work with signed and unsigned integer indices.
This also applies to record field-position indexing and to the get function for records and lists.
Apr 20, 2026 · @tobim, @codex · #6059
Partition rebuilds now finish after persisting rebuilt partitions. Previously, rebuild jobs could remain stuck indefinitely even though the replacement partitions were written successfully.
The summarize operator now starts frequency-based emission with the first
input event and emits overdue periodic results before later events are
aggregated. This makes periodic output deterministic in reset,
cumulative, and update modes for delayed or sparse streams.
For example:
from {ts: 0ms.from_epoch(), x: 1},
{ts: 90ms.from_epoch(), x: 1},
{ts: 360ms.from_epoch(), x: 1}
delay ts
summarize count=count(), options={frequency: 300ms, mode: "cumulative"}
The first periodic result now consistently reports a count of 2 before the
third event arrives.
Apr 16, 2026 · @tobim, @codex · #6039
Tenzir nodes now honor standard HTTP proxy environment variables when connecting to Tenzir Platform:
HTTPS_PROXY=http://proxy.example:3128 tenzir-node
Use NO_PROXY to bypass the proxy for selected hosts. This helps deployments where outbound connections to the Platform websocket gateway must go through an HTTP proxy.
Apr 16, 2026 · @mavam, @codex · #6022
The hash_* functions now hash blob values by their raw bytes. This makes checksums computed from binary data match external tools such as md5sum and sha256sum.
For example:
from_file "trace.pcap" {
read_all binary=true
}
md5 = data.hash_md5()
This is useful for verifying file contents and round-tripping binary formats without leaving TQL.
]]>context::lookup operator, and it adds pipeline names to diagnostics and metrics for easier operational correlation. This release also improves export reliability under load and fixes Azure transport errors, HTTP Host headers for non-standard ports, and rebuilt-partition export correctness.
Apr 1, 2026 · @IyeOnline · #5964
The context::lookup operator enables unified matching of events against contexts
by combining live and retrospective filtering in a single operation.
The operator automatically translates context updates into historical queries while simultaneously filtering all newly ingested data against any context updates.
This provides:
live=trueretro=trueExample usage:
context::lookup "feodo", field=src_ip
where @name == "suricata.flow"
Mar 30, 2026 · @IyeOnline, @claude · #5959
The metrics and diagnostics operators now include a pipeline_name field.
Previously, output from these operators only identified the source pipeline by its ID. Now the human-readable name is available too, making it straightforward to filter or group results by pipeline name without needing to look up IDs separately.
Please keep in mind that pipeline names are not unique.
Apr 8, 2026 · @claude
Bumped Apache Arrow from 23.0.0 to 23.0.1, which includes an upstream fix
for unhandled Azure::Core::Http::TransportException in Arrow's
AzureFileSystem methods. Previously, transport-level errors (e.g., SSL
certificate failures) could crash the node during file listing, reading, or
writing. Additionally, the direct Azure SDK calls in the blob deletion code
paths now catch Azure::Core::RequestFailedException (the common base of
both StorageException and TransportException) instead of listing
specific exception types.
Apr 7, 2026 · @tobim, @codex · #5988
The export operator no longer emits partially populated events from rebuilt partitions when a row is null at the record level. Previously, some events could appear with most fields set to null while a few values, such as event_type or interface fields, were still present.
This makes exports from rebuilt data more reliable when investigating sparse or malformed-looking events.
Mar 31, 2026
The from_http and http operators now include the port in the Host header
when the URL uses a non-standard port. Previously, the port was omitted, which
caused requests to fail with HTTP 403 when the server validates the Host
header against the full authority, such as for pre-signed URL signature
verification.
The export command no longer fails or misses recent events when a node is flushing active partitions to disk under heavy load. Recent exports now keep the in-memory partitions they depend on alive until the snapshot completes, which preserves correctness for concurrent import and export workloads.
Feb 4, 2026 · @tobim, @codex · #5703
AWS operators now support OIDC-based authentication via the AssumeRoleWithWebIdentity API.
You can authenticate with AWS resources using OpenID Connect tokens from external identity providers like Azure, Google Cloud, or custom endpoints. This enables secure cross-cloud authentication without sharing long-lived AWS credentials.
Configure web identity authentication in any AWS operator by specifying a token source and target role:
from_s3 "s3://bucket/path", aws_iam={
region: "us-east-1",
assume_role: "arn:aws:iam::123456789012:role/cross-cloud-role",
web_identity: {
token_file: "/path/to/oidc/token"
}
}
The web_identity option accepts three token sources: token_file (path to a token file), token_endpoint (HTTP endpoint that returns a token), or token (direct token value). For HTTP endpoints, you can extract tokens from JSON responses using path.
Credentials automatically refresh before expiration, with exponential backoff retry logic for transient failures. This is especially useful for long-running pipelines that need persistent authentication.
Mar 30, 2026 · @jachris · #5954
Pipelines that use and, or, or if-else expressions run significantly faster in certain cases — up to 30× in our benchmarks. The improvement is most noticeable in pipelines with complex filtering or branching logic. No pipeline changes are needed to benefit.
Mar 23, 2026 · @mavam, @codex · #5939
The ocsf::derive operator now supports OCSF 1.8.0 events.
For example, you can now derive enum and sibling fields for events that declare
metadata.version: "1.8.0":
from {metadata: {version: "1.8.0"}, class_uid: 1007}
ocsf::derive
This keeps OCSF normalization pipelines working when producers emit 1.8.0
events.
Platform configuration validation now provides clearer error messages when an invalid configuration is encountered, helping you quickly diagnose and fix configuration issues.
Setting TENZIR_ENDPOINT to an unresolvable hostname no longer crashes the pipeline with a segfault.
Mar 25, 2026 · @mavam, @claude · #5949
ocsf::derive no longer emits a false warning when an _id field is set
to 99 (Other) and the sibling string contains a source-specific value.
Per the OCSF specification, 99/Other is an explicit escape hatch: the
integer signals that the value is not in the schema's enumeration and the
companion string must hold the raw value from the data source. For
example, the following is now accepted silently:
from {
metadata: { version: "1.7.0" },
type_uid: 300201,
class_uid: 3002,
auth_protocol_id: 99,
auth_protocol: "Negotiate",
}
ocsf::derive
Previously this produced a spurious warning: found invalid value for 'auth_protocol' because "Negotiate" is not a named enum caption.
Mar 24, 2026 · @lava
The Azure Blob Storage connector now handles Azure::Core::Http::TransportException
(e.g., SSL certificate errors) gracefully instead of crashing. Previously, a
self-signed certificate in the certificate chain would cause an unhandled
exception and terminate the node.
Mar 17, 2026 · @tobim
Feather store files now include a TENZIR:store:origin key in the Arrow table
schema metadata. The value is "ingest" for freshly ingested data, "rebuild"
for partitions created by the rebuild command, and "compaction" for partitions
created by the compaction plugin. This allows external tooling such as pyarrow
to distinguish how a partition was produced.
Mar 11, 2026 · @IyeOnline, @codex, @mavam, @raxyte · #5897
The to_clickhouse operator now supports dynamic table names via an expression
table=..., which must evaluate to a string. If the value is not a valid
table name, the events will be dropped with a warning.
With this change, the operator will also create a database if it does not exist.
The prime use-case for this are OCSF event streams:
subscribe "ocsf"
ocsf::cast encode_variants=true, null_fill=true
to_clickhouse table=f"ocsf.{class_name.replace(" ","_")}", ...
You can now install Tenzir on Apple Silicon macOS via Homebrew:
brew tap tenzir/tenzir
brew install --cask tenzir
You can also install directly without tapping first:
brew install --cask tenzir/tenzir/tenzir
The release workflow keeps the Homebrew cask in sync with the signed macOS package so installs and uninstalls stay current across releases.
The AWS Marketplace ECR repository tenzir-node was incorrectly populated with
the tenzir image. It now correctly ships tenzir-node, which runs a Tenzir
node by default.
If you relied on the previous behavior, you can restore it by setting tenzir
as a custom entrypoint in your ECS task definition.
The bundled Suricata schema now covers the remaining event types listed in the Suricata 8.0.3 EVE JSON format documentation: IKE (IKEv1/IKEv2), HTTP/2, PostgreSQL, and Modbus. This completes Suricata 8 schema coverage for Tenzir.
The read_syslog operator and parse_syslog function now accept RFC 5424 structured-data parameter names longer than 32 characters, which some vendors emit despite the specification limit.
For example, this message now parses successfully instead of being rejected:
<134>1 2026-03-18T11:00:51.194137+01:00 HOSTNAME abc 9043 23003147 [F5@12276 thx_f5_for_ignoring_the_32_char_limit_in_structured_data="thx"] broken example
This improves interoperability with vendor syslog implementations that exceed the RFC limit for structured-data parameter names.
Mar 14, 2026 · @aljazerzen · #5906
The batch timeout was only checked when a new event arrived, so a single event followed by an idle stream would never be emitted. The timeout now fires independently of upstream activity.
Mar 13, 2026 · @jachris · #5901
The parse_winlog function could fragment output into thousands of tiny
batches due to type conflicts in RenderingInfo/Keywords, where events with
one <Keyword> emitted a string but events with multiple emitted a list.
Additionally, EventData with unnamed <Data> elements is now always emitted
as a record with _0, _1, etc. as field names instead of a list.
Mar 12, 2026 · @jachris · #5899
The in operator for list expressions is up to 33x faster. Previously it
created and finalized entire Arrow arrays for every element comparison, causing
severe overhead for expressions like EventID in [5447, 4661, ...].
Additionally, comparing a typed null value with == now returns false instead
of null, and != returns true, fixing a correctness issue with null
handling in equality comparisons.
Mar 12, 2026 · @jachris · #5899
The in operator fast path now correctly prevents comparison of secret values.
Previously, secret_value in [...] would silently compare instead of returning
null with a warning, bypassing the established secret comparison policy.
Mar 12, 2026 · @jachris · #5900
Pattern equality checks now correctly consider the case-insensitive flag. Previously, two patterns that differed only in case sensitivity were treated as equal, violating the hash/equality contract.
Mar 12, 2026 · @jachris · #5899
Splitting Arrow arrays for string and blob types no longer over-reserves memory. Previously both output builders reserved the full input size each, using up to twice the necessary memory.
]]>Fixed a scheduling issue introduced in v5.24.0 that could cause the node to
become unresponsive when too many pipelines using detached operators like
from_udp were deployed simultaneously.
The bundled Suricata schema now includes types for four previously missing event types: ike, http2, pgsql, and modbus.
The ike type supports both IKEv1 and IKEv2 traffic. Version-specific fields are contained within dedicated ikev1 and ikev2 sub-objects, covering key exchange payloads, nonce payloads, client proposals, vendor IDs, and IKEv2 role/notify information.
The http2 type models HTTP/2 request and response streams including settings frames, header lists, error codes, and stream priority.
The pgsql type covers PostgreSQL session events with full request fields (simple queries, startup parameters, SASL authentication) and response fields (row counts, command completion, parameter status).
The modbus type captures industrial Modbus protocol transactions including function codes, access types, exception responses, diagnostic subfunctions, and MEI encapsulated interface data.
Mar 13, 2026 · @mavam, @codex · #5902
read_syslog and parse_syslog now extract a leading RFC 5424-style
structured-data block from RFC 3164 message content.
This pattern occurs in practice with some VMware ESXi messages, where
components such as Hostd emit a legacy syslog record and prepend structured
metadata before the human-readable message text.
For example, this raw syslog line:
<166>2026-02-11T18:01:45.587Z esxi-01.example.invalid Hostd[2099494]: [Originator@6876 sub=Vimsvc.TaskManager opID=11111111-2222-3333-4444-555555555555] Task Completed
now parses as:
{
facility: 20,
severity: 6,
timestamp: "2026-02-11T18:01:45.587Z",
hostname: "esxi-01.example.invalid",
app_name: "Hostd",
process_id: "2099494",
structured_data: {
"Originator@6876": {
sub: "Vimsvc.TaskManager",
opID: "11111111-2222-3333-4444-555555555555",
},
},
content: "Task Completed",
}
Events without extracted structured data keep the existing syslog.rfc3164
schema. Events with extracted structured data use
syslog.rfc3164.structured.
Mar 10, 2026 · @IyeOnline, @satta · #5888
The bundled Suricata schema now aligns with Suricata 8, enabling proper parsing and representation of events from Suricata 8 deployments.
This update introduces support for new event types including POP3, ARP, and BitTorrent DHT, along with enhancements to existing event types. QUIC events now include ja4 and ja4s fields for fingerprinting, DHCP events include vendor_class_identifier, and TLS certificate timestamps now use the precise time type instead of string representation.
These schema changes ensure that Tenzir can reliably ingest and process telemetry from Suricata 8 without data loss or type mismatches.
Mar 11, 2026 · @jachris · #5893
In some situations, pipelines could not be successfully started, leading to timeouts and a non-responsive node, especially during node start.
Mar 10, 2026 · @IyeOnline, @codex · #5886
Pipelines using chained list transforms such as xs.where(...).map(...).where(...) no longer trigger an internal assertion on sliced input batches.
Mar 9, 2026 · @mavam, @codex · #5877
Invalid Google Cloud credentials in from_google_cloud_pubsub no longer crash the node. Authentication errors now surface as operator diagnostics instead.
Mar 2, 2026 · @mavam, @codex · #5851
parse_syslog() and read_syslog now accept common Check Point structured-data variants that are not strictly RFC 5424 compliant. This includes key:"value" parameters, semicolon-separated parameters, and records that omit an SD-ID entirely.
For records without an SD-ID, Tenzir now normalizes the structured data under checkpoint_2620, so downstream pipelines can use a stable field path.
For example, the message <134>1 ... - [action:"Accept"; conn_direction:"Incoming"] now parses successfully and maps to structured_data.checkpoint_2620. This improves interoperability with Check Point exports and reduces ingestion-time preprocessing.
Mar 6, 2026 · @IyeOnline · #5805
JSON parsing errors now display the surrounding bytes at the error location. This makes it easier to diagnose malformed JSON in your data pipelines.
For example, if your JSON is missing a closing bracket, the error message shows you the bytes around that location and marks where the parser stopped expecting more input.
Mar 4, 2026 · @tobim, @codex · #5865
The load_tcp operator now makes DNS hostname resolution opt-in with the resolve_hostnames parameter (defaults to false).
Previously, the operator always attempted reverse DNS lookups for peer endpoints, which could fail in environments without working reverse DNS configurations. Now you can enable this behavior by setting resolve_hostnames to true:
load_tcp endpoint="0.0.0.0:5555" resolve_hostnames=true {
read_json
}
When enabled and DNS resolution fails, the operator emits a warning diagnostic (once) instead of failing. This allows the operator to continue functioning in environments where reverse DNS is unavailable or unreliable.
Mar 6, 2026 · @IyeOnline · #5805
We improved the reporting for unexpected diagnostics outside of operator execution, such as during startup. In these cases you will now get the diagnostic message.
Mar 6, 2026 · @IyeOnline · #5805
Fixed multiple issues that could cause errors or incorrect behavior when the schema of parsed events changes between records. This is particularly important when ingesting data from sources that may add, remove, or modify fields over time.
Schema mismatch warnings for repeated fields in JSON objects (which Tenzir interprets as lists) now include an explanatory hint, making it clearer what's happening when a field appears multiple times where a single value was expected.
]]>Mar 2, 2026 · @IyeOnline · #5855
We fixed a bug that could cause a crash when reading JSON data.
Mar 2, 2026 · @jachris · #5841
The CEF parser now handles unescaped = characters (which are not conforming to
the specification) by using a heuristic.
Feb 27, 2026 · @mavam, @codex · #5846
The new experimental hmac function computes Hash-based Message
Authentication Codes (HMAC) for strings and blobs. It supports SHA-256
(default), SHA-512, SHA-384, SHA-1, and MD5 algorithms.
Note: The key parameter is currently a plain string because function
arguments cannot be secrets yet. We plan to change this in the future.
from {
signature: hmac("hello world", "my-secret-key"),
}
{
signature: "90eb182d8396f16d4341d582047f45c0a97d73388c5377d9ced478a2212295ad",
}
Specify a different algorithm with the algorithm parameter:
from {
signature: hmac("hello world", "my-secret-key", algorithm="sha512"),
}
Feb 27, 2026 · @IyeOnline · #5842
We fixed a bug that would cause an assertion failure "Index error: array slice would exceed array length". This was introduced as part of an optimization in Tenzir Node v5.27.0.
]]>Feb 24, 2026 · @lava
Fixed in issue where the platform plugin did not correctly use the
configured certfile, keyfile and cafile options for client
certificate authentication, and improved the error messages for TLS
issues during platform connection.