# Tenzir Node v4.12.0 ## 馃殌 Features ### PRs 4133-4138-satta Apr 22, 2024 路 [@satta](https://github.com/satta) 路 [#4133](https://github.com/tenzir/tenzir/pull/4133) The `suricata` parser鈥檚 schema now more accurately reflects Suricata鈥檚 Eve JSON output, adding many fields that were previously missing. ### Add `show schemas` to display all available schemas Apr 19, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4131](https://github.com/tenzir/tenzir/pull/4131) The `show schemas` operator lists all unique schemas of events stored at the node. ### Add configurable contexts Apr 18, 2024 路 [@tobim](https://github.com/tobim) 路 [#4126](https://github.com/tenzir/tenzir/pull/4126) You can now define contexts and their creation parameters in the `tenzir.contexts` section of the configuration file. ### Add a delay to retrying failed pipelines Apr 17, 2024 路 [@Dakostu](https://github.com/Dakostu) 路 [#4108](https://github.com/tenzir/tenzir/pull/4108) Stopping a failed pipeline now moves it into the stopped state in the app and through the `/pipeline/update` API, stopping automatic restarts on failure. Pipelines now restart on failure at most every minute. The new API parameter `retry_delay` is available in the `/pipeline/create`, `/pipeline/launch`, and `/pipeline/update` APIs to customize this value. For configured pipelines, the new `restart-on-error` option supersedes the previous `autostart.failed` option and may be set either to a boolean or to a duration, with the former using the default retry delay and the latter using a custom one. The output of `show pipelines` and the `/pipeline/list` API now includes the start time of the pipeline in the field `start_time`, the newly added retry delay in the field `retry_delay`, and whether the pipeline is hidden from the overview page on app.tenzir.com in the field `hidden`. ### Add value grouping to `chart` and remove `--title` Apr 15, 2024 路 [@jachris](https://github.com/jachris) 路 [#4119](https://github.com/tenzir/tenzir/pull/4119) Some charts supported by the `chart` operator (`bar`, `line`, and `area`) now have a `--position` argument, with the possible values of `grouped` and `stacked`. ### Support 0mq inproc sockets Apr 12, 2024 路 [@mavam](https://github.com/mavam) 路 [#4117](https://github.com/tenzir/tenzir/pull/4117) The `0mq` connector now supports `inproc` socket endpoint URLs, allowing you to create arbitrary publish/subscribe topologies within a node. For example, `save zmq inproc://foo` writes messages to the in-process socket named `foo`. ### Generalize `every` to work with all operators Apr 12, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4109](https://github.com/tenzir/tenzir/pull/4109) The `every ` operator modifier now supports all operators, turning blocking operators like `tail`, `sort` or `summarize` into operators that emit events every ``. ### Add multi-line syslog message support Apr 8, 2024 路 [@eliaskosunen](https://github.com/eliaskosunen) 路 [#4080](https://github.com/tenzir/tenzir/pull/4080) Syslog messages spanning multiple lines are now supported. ### Add a `--timeout ` option to `batch` Apr 5, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4095](https://github.com/tenzir/tenzir/pull/4095) The `batch` operator gained a new `--timeout ` option that controls the maixmum latency for withholding events for batching. ### Add `unroll` operator Mar 28, 2024 路 [@jachris](https://github.com/jachris) 路 [#4078](https://github.com/tenzir/tenzir/pull/4078) The `unroll` operator transforms an event that contains a list into a sequence of events where each output event contains one of the list elements. ### Introduce the BITZ format Mar 27, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4079](https://github.com/tenzir/tenzir/pull/4079) The `bitz` format resembles Tenzir鈥檚 internal wire format. It enables lossless and quick transfer of events between Tenzir nodes through any connector. ### Add UDP connector Mar 27, 2024 路 [@mavam](https://github.com/mavam) 路 [#4067](https://github.com/tenzir/tenzir/pull/4067) The new `udp` connector comes with a loader and saver to read bytes from and write bytes to a UDP socket. ### Add `deduplicate` operator Mar 27, 2024 路 [@eliaskosunen](https://github.com/eliaskosunen) 路 [#4068](https://github.com/tenzir/tenzir/pull/4068) The `deduplicate` operator allows removing duplicate events based on specific fields. ## 馃敡 Changes ### Remove events output from many context operators Apr 23, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4143](https://github.com/tenzir/tenzir/pull/4143) The `context create`, `context reset`, `context update`, and `context load` operators no return information about the context. Pipelines ending with these operators will now be considered closed, and you will be asked to deploy them in the Explorer. Previously, users commonly added `discard` after these operators to force this behavior. ### Implement a `parquet` parser and printer Apr 22, 2024 路 [@balavinaithirthan](https://github.com/balavinaithirthan) 路 [#4116](https://github.com/tenzir/tenzir/pull/4116) The `parquet` format more efficiently reads and writes Parquet files. The format is streamable for `write parquet`. ### Add value grouping to `chart` and remove `--title` Apr 15, 2024 路 [@jachris](https://github.com/jachris) 路 [#4119](https://github.com/tenzir/tenzir/pull/4119) In the `chart` operator, unless otherwise specified, every field but the first one is taken to be a value for the Y-axis, instead of just the second one. If the value for `-x`/`--name` or `-y`/`--value` is explicitly specified, the other one must now be too. The `--title` option is removed from `chart`. Titles can instead be set directly in the web interface. ### Support 0mq inproc sockets Apr 12, 2024 路 [@mavam](https://github.com/mavam) 路 [#4117](https://github.com/tenzir/tenzir/pull/4117) The `0mq` connector no longer automatically monitors TCP sockets to wait until at least one remote peer is present. Explicitly pass `--monitor` for this behavior. ### Remove metrics from `/pipeline/list` Apr 12, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4114](https://github.com/tenzir/tenzir/pull/4114) The `show pipelines` operator and `/pipeline/list` endpoint no longer include pipeline metrics. We recommend using the `metrics` operator instead, which offers the same data in a more flexible way. ### Implement feather printer and parser Apr 10, 2024 路 [@balavinaithirthan](https://github.com/balavinaithirthan) 路 [#4089](https://github.com/tenzir/tenzir/pull/4089) The `feather` format now reads and writes Arrow IPC streams in addition to Feather files, and no longer requires random access to a file to function, making the format streamable with both `read feather` and `write feather`. ### Add multi-line syslog message support Apr 8, 2024 路 [@eliaskosunen](https://github.com/eliaskosunen) 路 [#4080](https://github.com/tenzir/tenzir/pull/4080) Lines of input containing an invalid syslog messages are now assumed to be a continuation of a message on a previous line, if there鈥檚 any. ### Remove many deprecated things Apr 7, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4103](https://github.com/tenzir/tenzir/pull/4103) The `tenzir-ctl count ` command no longer exists. It has long been deprecated and superseded by pipelines of the form `export | where | summarize count(.)`. The deprecated `tenzir-ctl status` command and the corresponding `/status` endpoint no longer exist. They have been superseded by the `show` and `metrics` operators that provide more detailed insight. The deprecated `tenzir.aging-frequency` and `tenzir.aging-query` options no longer exist. We recommend using the compaction or disk monitor mechanisms instead to delete persisted events. ## 馃悶 Bug Fixes ### Fix HTTP saver Content-Length computation Apr 23, 2024 路 [@mavam](https://github.com/mavam) 路 [#4134](https://github.com/tenzir/tenzir/pull/4134) The `http` saver now correctly sets the `Content-Length` header value for HTTP POST requests. ### Remove events output from many context operators Apr 23, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4143](https://github.com/tenzir/tenzir/pull/4143) The `enrich` operator sometimes stopped working when it encountered an event for which the specified fields did not exist. This no longer happens. ### Fix verification of large FlatBuffers tables Apr 22, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4137](https://github.com/tenzir/tenzir/pull/4137) Lookup tables with more than 1M entries failed to load after the node was restarted. This no longer happens. ### Fix a race condition in `/serve` Apr 16, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4123](https://github.com/tenzir/tenzir/pull/4123) We fixed a bug that caused the explorer to sometimes show 504 Gateway Timeout errors for pipelines where the first result took over two seconds to arrive. ### Support parallel connections in `from tcp` Apr 15, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4084](https://github.com/tenzir/tenzir/pull/4084) The `tcp` connector now supports accepting multiple connections in parallel when used with the `from` operator, parsing data separately per connection. ### Fix unflattening of empty records and `null` records Apr 9, 2024 路 [@jachris](https://github.com/jachris) 路 [#4104](https://github.com/tenzir/tenzir/pull/4104) Empty records and `null` values of record type are now correctly unflattened. ### Shut down node a configured pipeline fails to start Apr 5, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4097](https://github.com/tenzir/tenzir/pull/4097) Nodes now shut down with a non-zero exit code when pipelines configured as part of the `tenzir.yaml` file fail to start, making such configuration errors easier to spot. ### Remove wrong EXPOSE in Dockerfile Apr 5, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4099](https://github.com/tenzir/tenzir/pull/4099) Tenzir Docker images no longer expose 5158/tcp by default, as this prevented running multiple containers in the same network or in host mode. ### Fix shutdown of connected pipelines alongside node Apr 5, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4093](https://github.com/tenzir/tenzir/pull/4093) Pipelines run with the `tenzir` binary that connected to a Tenzir Node did sometimes not shut down correctly when the node shut down. This now happens reliably. ### Fix HTTP PUT with empty request body Apr 4, 2024 路 [@mavam](https://github.com/mavam) 路 [#4092](https://github.com/tenzir/tenzir/pull/4092) We fixed a bug in the `http` saver that prevented sending HTTP PUT requests with an empty request body. ### Make `python` operator not discard fields that start with an underscore Apr 4, 2024 路 [@jachris](https://github.com/jachris) 路 [#4085](https://github.com/tenzir/tenzir/pull/4085) The `python` operator no longer discards field that start with an underscore. ### Fix `python` deadlock for empty input Apr 4, 2024 路 [@jachris](https://github.com/jachris) 路 [#4086](https://github.com/tenzir/tenzir/pull/4086) The `python` operator no longer deadlocks when given an empty program. ### Fix JSON printer handling of `inf` and `nan` Apr 3, 2024 路 [@jachris](https://github.com/jachris) 路 [#4087](https://github.com/tenzir/tenzir/pull/4087) The JSON printer previously printed invalid JSON for `inf` and `nan`, which means that `serve` could sometimes emit invalid JSON, which is not handled well by platform/app. Instead, we now emit `null`. ### Fix the schema name in `show contexts` Mar 28, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4082](https://github.com/tenzir/tenzir/pull/4082) The schema name of events returned by `show contexts` sometimes did not match the type of the context. This now works reliably. [ Download on GitHub ](https://github.com/tenzir/tenzir/releases/tag/v4.12.0) [Get the release artifacts and source code.](https://github.com/tenzir/tenzir/releases/tag/v4.12.0)