# Tenzir Node v3.1.0

## 🚀 Features

### Implement a `zeek-tsv` format

May 11, 2023 · [@Dakostu](https://github.com/Dakostu) · [#3114](https://github.com/tenzir/tenzir/pull/3114)

The `zeek-tsv` format parses and prints Zeek’s native tab-separated value (TSV) representation of logs.

### Implement a distributed pipeline executor

May 6, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3119](https://github.com/tenzir/tenzir/pull/3119)

Pipelines may now span across multiple processes. This will enable upcoming operators that do not just run locally in the `vast exec` process, but rather connect to a VAST node and partially run in that node. The new operator modifiers `remote` and `local` allow expert users to control where parts of their pipeline run explicitly, e.g., to offload compute to a more powerful node. Potentially unsafe use of these modifiers requires setting `vast.allow-unsafe-pipelines` to `true` in the configuration file.

### Introduce a `version` source operator

May 5, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3123](https://github.com/tenzir/tenzir/pull/3123)

The `vast exec` command now supports implicit sinks for pipelines that end in events or bytes: `write json --pretty` and `save file -`, respectively.

The `--pretty` option for the JSON printer enables multi-line output.

The new `version` source operator yields a single event containing VAST’s version and a list of enabled plugins.

### Implement the `measure` operator

May 3, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3093](https://github.com/tenzir/tenzir/pull/3093)

The `inspect` operator replaces the events or bytes it receives with incremental metrics describing the input.

### Add new CEF parser plugin

Apr 29, 2023 · [@jachris](https://github.com/jachris) · [#3110](https://github.com/tenzir/tenzir/pull/3110)

The `cef` parser allows for using the CEF format with the new pipelines.

### Add Feather and Parquet parsers and printers

Apr 28, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3103](https://github.com/tenzir/tenzir/pull/3103)

The `feather` and `parquet` formats allow for reading and writing events from and to the Apache Feather V2 and Apache Parquet files, respectively.

### Implement `xsv` parser & printer

Apr 27, 2023 · [@Dakostu](https://github.com/Dakostu) · [#3104](https://github.com/tenzir/tenzir/pull/3104)

The `xsv` format enables the user to parse and print character-separated values, with the additional `csv`, `tsv` and `ssv` formats as sane defaults.

### Add `directory` saver

Apr 25, 2023 · [@Dakostu](https://github.com/Dakostu) · [#3098](https://github.com/tenzir/tenzir/pull/3098)

The new `directory` sink creates a directory with a file for each schema in the specified format.

### PRs 3085-3088-3097

Apr 20, 2023 · [@jachris](https://github.com/jachris) · [#3085](https://github.com/tenzir/tenzir/pull/3085)

The new `file` connector enables the user to process file input/output as data in a pipeline. This includes regular files, UDS files as well as `stdin/stdout`.

### Upgrade exporter to use new pipelines

Apr 19, 2023 · [@jachris](https://github.com/jachris) · [#3076](https://github.com/tenzir/tenzir/pull/3076)

The `vast export` command now accepts the new pipelines as input. Furthermore, `vast export <expr>` is now deprecated in favor of `vast export 'where <expr>'`.

### Expose the lower-level `load`, `parse`, `print`, and `save` operators

Apr 17, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3079](https://github.com/tenzir/tenzir/pull/3079)

The new `from <connector> [read <format>]`, `read <format> [from <connector>]`, `write <format> [to <connector>]`, and `to <connector> [write <format>]` operators bring together a connector and a format to prduce and consume events, respectively. Their lower-level building blocks `load <connector>`, `parse <format>`, `print <format>`, and `save <connector>` enable expert users to operate on raw byte streams directly.

### Upgrade partition transformer to new pipelines

Apr 13, 2023 · [@jachris](https://github.com/jachris) · [#3064](https://github.com/tenzir/tenzir/pull/3064)

User-defined operator aliases make pipelines easier to use by enabling users to encapsulate a pipelinea into a new operator. To define a user-defined operator alias, add an entry to the `vast.operators` section of your configuration.

Compaction now makes use of the new pipeline operators, and allows pipelines to be defined inline instead in addition to the now deprecated `vast.pipelines` configuration section.

### Introduce the `count_distinct` aggregation function

Apr 12, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3068](https://github.com/tenzir/tenzir/pull/3068)

The `count_distinct` aggregation function returns the number of distinct, non-null values.

### Add `unique` operator

Apr 12, 2023 · [@jachris](https://github.com/jachris) · [#3051](https://github.com/tenzir/tenzir/pull/3051)

The newly-added `unique` operator removes adjacent duplicates.

### Add `tail` operator

Apr 2, 2023 · [@Dakostu](https://github.com/Dakostu) · [#3050](https://github.com/tenzir/tenzir/pull/3050)

The new `tail` pipeline operator limits all latest events to a specified number. The operator takes the limit as an optional argument, with the default value being 10.

### PRs 3036-3039-3089

Mar 29, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3036](https://github.com/tenzir/tenzir/pull/3036)

The `put` operator is the new companion to the existing `extend` and `replace` operators. It specifies the output fields exactly, referring either to input fields with an extractor, metadata with a selector, or a fixed value.

The `extend` and `replace` operators now support assigning extractors and selectors in addition to just fixed values.

## 🔧 Changes

### Upgrade exporter to use new pipelines

Apr 19, 2023 · [@jachris](https://github.com/jachris) · [#3076](https://github.com/tenzir/tenzir/pull/3076)

The `exporter.*` metrics no longer exist, and will return in a future release as a more generic instrumentation mechanism for all pipelines.

### Add support for user-defined operator aliases

Apr 12, 2023 · [@jachris](https://github.com/jachris) · [#3067](https://github.com/tenzir/tenzir/pull/3067)

The `vast.operators` section in the configuration file supersedes the now deprecated `vast.pipelines` section and more generally enables user-defined operators. Defined operators now must use the new, textual format introduced with VAST v3.0, and are available for use in all places where pipelines are supported.

### Restart the systemd service on failure

Apr 12, 2023 · [@tobim](https://github.com/tobim) · [#3058](https://github.com/tenzir/tenzir/pull/3058)

The bundled systemd service is now configured to restart VAST in case of a failure.

### Remove configuration-defined import/export pipelines

Apr 3, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3052](https://github.com/tenzir/tenzir/pull/3052)

As already announced with the VAST v3.0 release, the `vast.pipeline-triggers` option now no longer functions. The feature will be replaced with node ingress/egress pipelines that fit better into a multi-node model than the previous feature that was built under the assumption of a client/server model with a single server.

### Update query endpoint to use new pipeline executor

Mar 31, 2023 · [@jachris](https://github.com/jachris) · [#3015](https://github.com/tenzir/tenzir/pull/3015)

The `/query` REST endpoint no longer accepts an expression at the start of the query. Instead, use `where <expr> | ...`.

## 🐞 Bug Fixes

### Fix remaining partitions counter in the rebuilder

May 12, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3147](https://github.com/tenzir/tenzir/pull/3147)

The `rebuilder.partitions.remaining` metric sometimes reported wrong values when partitions for at least one schema did not need to be rebuilt. We aligned the metrics with the actual functionality.

### Set minimum timestamp of partitions properly

May 11, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3141](https://github.com/tenzir/tenzir/pull/3141)

Some pipelines in compaction caused transformed partitions to be treated as if they were older than they were supposed to be, causing them to be picked up again for deletion too early. This bug no longer exists, and compacted partitions are now considered at most as old as the oldest event before compaction.

### Align endpoints between regular and slim Docker images

May 10, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3137](https://github.com/tenzir/tenzir/pull/3137)

The `tenzir/vast` image now listens on `0.0.0.0:5158` instead of `127.0.0.1:5158` by default, which aligns the behavior with the `tenzir/vast-slim` image.

### Bump vast-plugins to a95e420

May 4, 2023 · [@tobim](https://github.com/tobim) · [#3115](https://github.com/tenzir/tenzir/pull/3115)

The matcher plugin no longer causes deadlocks through detached matcher clients.

### Mark some CAF types as nodiscard

Apr 19, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3086](https://github.com/tenzir/tenzir/pull/3086)

Tokens created with `vast web generate-token` now persist correctly, and work across restarts of VAST.

### Introduce the `count_distinct` aggregation function

Apr 12, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3068](https://github.com/tenzir/tenzir/pull/3068)

The `distinct` function silently performed a different operation on lists, returning the distinct non-null elements in the list rather than operating on the list itself. This special-casing no longer exists, and instead the function now operates on the lists itself. This feature will return in the future as unnesting on the extractor level via `distinct(field[])`, but for now it has to go to make the `distinct` aggregation function work consistently.

### Fix subnet queries for some subnets

Apr 6, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3060](https://github.com/tenzir/tenzir/pull/3060)

VAST incorrectly handled subnets using IPv6 addresses for which an equivalent IPv4 address existed. This is now done correctly. For example, the query `where :ip !in ::ffff:0:0/96` now returns all events containing an IP address that cannot be represented as an IPv4 address. As an additional safeguard, the VAST language no longer allows for constructing subnets for IPv4 addresses with lengths greater than 32.

[ Download on GitHub ](https://github.com/tenzir/tenzir/releases/tag/v3.1.0)

[Get the release artifacts and source code.](https://github.com/tenzir/tenzir/releases/tag/v3.1.0)