# Tenzir Node v3.0.0

## 🚀 Features

### Support pattern case insensitivity in Sigma plugin

Mar 3, 2023 · [@Dakostu](https://github.com/Dakostu) · [#2974](https://github.com/tenzir/tenzir/pull/2974)

The `sigma` plugin now treats Sigma strings as case-insensitive patterns during the transpilation process.

### Add extractor predicates

Mar 2, 2023 · [@jachris](https://github.com/jachris) · [#2984](https://github.com/tenzir/tenzir/pull/2984)

Extractors such as `x` and `:T` can now expand to the predicates `x != null` and `:T != null`, respectively.

### Add a new /openapi.json endpoint to the web plugin

Mar 2, 2023 · [@lava](https://github.com/lava) · [#2981](https://github.com/tenzir/tenzir/pull/2981)

The experimental web plugin now serves its own API specification at the new ‘/openapi.json’ endpoint.

### Support case insensitivity in patterns

Feb 17, 2023 · [@Dakostu](https://github.com/Dakostu) · [#2951](https://github.com/tenzir/tenzir/pull/2951)

Patterns now support case insensitivity by adding `i` to the pattern string, e.g. `/^\w{3}$/i`.

### Add CORS preflight request handling

Feb 16, 2023 · [@lava](https://github.com/lava) · [#2944](https://github.com/tenzir/tenzir/pull/2944)

The experimental web frontend now correctly responds to CORS preflight requests. To configure CORS behavior, the new `vast.web.cors-allowed-origin` config option can be used.

### Implement `head` and `taste` operators

Jan 30, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2891](https://github.com/tenzir/tenzir/pull/2891)

The new `head` and `taste` operators limit results to the specified number of events. The `head` operator applies this limit for all events, and the `taste` operator applies it per schema. Both operators take the limit as an optional argument, with the default value being 10.

### PRs 2877-2904-2907

Jan 27, 2023 · [@Dakostu](https://github.com/Dakostu) · [#2877](https://github.com/tenzir/tenzir/pull/2877)

The `export` and `import` commands now support an optional pipeline string that allows for chaining pipeline operators together and executing such a pipeline on outgoing and incoming data. This feature is experimental and the syntax is subject to change without notice. New operators are only available in the new pipeline syntax, and the old YAML syntax is deprecated.

### Implement a retry mechanism for VAST clients failing to connect to the server

Jan 23, 2023 · [@patszt](https://github.com/patszt) · [#2835](https://github.com/tenzir/tenzir/pull/2835)

We changed VAST client processes to attempt connecting to a VAST server multiple times until the configured connection timeout (`vast.connection-timeout`, defaults to 5 minutes) runs out. A fixed delay between connection attempts (`vast.connection-retry-delay`, defaults to 3 seconds) ensures that clients to not stress the server too much. Set the connection timeout to zero to let VAST client attempt connecting indefinitely, and the delay to zero to disable the retry mechanism.

### Add options to omit empty values when exporting as JSON

Jan 17, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2856](https://github.com/tenzir/tenzir/pull/2856)

The JSON export format gained the options `--omit-empty-records`, `--omit-empty-lists`, and `--omit-empty-maps`, which cause empty records, lists, and maps not to be rendered respectively. The options may be combined together with the existing `--omit-nulls` option. Use `--omit-empty` to set all four flags at once.

### Install Python bindings along with VAST

Jan 13, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2636](https://github.com/tenzir/tenzir/pull/2636)

VAST installations and packages now include Python bindings in a site-package under `<install-prefix>/lib/python*/site-packages/vast`.

### Add ‘pipeline’ parameter and schematized format to export endpoint

Jan 5, 2023 · [@lava](https://github.com/lava) · [#2773](https://github.com/tenzir/tenzir/pull/2773)

The `/export` family of endpoints now accepts an optional `pipeline` parameter to specify an ad-hoc pipeline that should be applied to the exported data.

### Make it easy to create docker images with Nix

Dec 17, 2022 · [@tobim](https://github.com/tobim) · [#2742](https://github.com/tenzir/tenzir/pull/2742)

We now offer a `tenzir/vast-slim` image as an alternative to the `tenzir/vast` image. The image is minimal in size and supports the same features as the regular image, but does not support building additional plugins against it and mounting in additional plugins.

### Add CEF reader plugin

Dec 13, 2022 · [@mavam](https://github.com/mavam) · [#2216](https://github.com/tenzir/tenzir/pull/2216)

The `cef` import format allows for reading events in the Common Event Format (CEF) via `vast import cef < cef.log`.

### Add `vast import arrow` enabling ingestion of arrow IPC format

Dec 12, 2022 · [@dispanser](https://github.com/dispanser) · [#2707](https://github.com/tenzir/tenzir/pull/2707)

VAST now imports Arrow IPC data, which is the same format it already supports for export.

### Add ‘pseudonymize’ pipeline operator

Dec 10, 2022 · [@Dakostu](https://github.com/Dakostu) · [#2719](https://github.com/tenzir/tenzir/pull/2719)

The new `pseudonymize` pipeline operator pseudonymizes IP addresses in user-specified fields.

### Implement support for `:string == /pattern/` queries

Dec 9, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2769](https://github.com/tenzir/tenzir/pull/2769)

Queries of the forms `:string == /pattern/`, `field == /pattern/`, `#type == /pattern/`, and their respective negations now work as expected.

### Eliminate shutdown lag from the signal monitor

Dec 6, 2022 · [@tobim](https://github.com/tobim) · [#2766](https://github.com/tenzir/tenzir/pull/2766)

The new `/query` endpoint for the experimental REST API allows users to receive query data in multiple steps, as opposed to a oneshot export.

## 🔧 Changes

### Deprecate `vast.pipeline-triggers`

Mar 10, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3008](https://github.com/tenzir/tenzir/pull/3008)

The `vast.pipeline-triggers` option is deprecated; while it continues to work as-is, support for it will be removed in the next release. Use the new inline import and export pipelines instead. They will return as more generally applicable node ingress and egress pipelines in the future.

### Replace ‘nil’ with ‘null

Mar 10, 2023 · [@Dakostu](https://github.com/Dakostu) · [#2999](https://github.com/tenzir/tenzir/pull/2999)

The non-value literal in expressions has a new syntax: `null` replaces its old representation `nil`. For example, the query `x != nil` is no longer valid; use `x != null` instead.

### Switch default TCP port to 5158

Mar 8, 2023 · [@tobim](https://github.com/tobim) · [#2998](https://github.com/tenzir/tenzir/pull/2998)

From now on VAST will use TCP port 5158 for its native inter process communication. This change avoids collisions from dynamic port allocation on Linux systems.

### Remove the /export endpoint

Mar 7, 2023 · [@Dakostu](https://github.com/Dakostu) · [#2990](https://github.com/tenzir/tenzir/pull/2990)

The REST API does not contain the `/export` and `/export/with-schemas` endpoints anymore. Any previous queries using those endpoints have to be sent to the `/query` endpoint now.

### Rename `identity` operator to `pass`

Mar 1, 2023 · [@jachris](https://github.com/jachris) · [#2980](https://github.com/tenzir/tenzir/pull/2980)

We renamed the `identity` operator to `pass`.

### Make the map type inaccessible to users

Mar 1, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2976](https://github.com/tenzir/tenzir/pull/2976)

The `map` type no longer exists: instead of `map<T, U>`, use the equivalent `list<record{ key: T, value: U }>`.

### Provide an Ansible role for VAST

Feb 9, 2023 · [@tobim](https://github.com/tobim) · [#2604](https://github.com/tenzir/tenzir/pull/2604)

VAST now comes with a role definition for Ansible. You can find it directly in the `ansible` subdirectory.

### PRs 2922-2927

Feb 8, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2922](https://github.com/tenzir/tenzir/pull/2922)

We removed the frontend prototype bundled with the web plugin Some parts of the frontend that we have in development are designed to be closed-source, and it is easier to develop at the current development stage in a single repository that is not bound to the release process of VAST itself. An open-source version of the frontend may return in the future.

### Use CAF streaming in export command

Feb 2, 2023 · [@patszt](https://github.com/patszt) · [#2898](https://github.com/tenzir/tenzir/pull/2898)

The `explore` and `pivot` commands are now unavailable. They will be reintroduced as pipeline operators in the future.

### Update response format of the /export endpoint

Feb 1, 2023 · [@lava](https://github.com/lava) · [#2899](https://github.com/tenzir/tenzir/pull/2899)

For the experimental REST API, the result format of the `/export` endpoint was modified: The `num_events` key was renamed to `num-events`, and the `version` key was removed.

### Align output of the Zeek TSV reader with schemas

Jan 28, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2887](https://github.com/tenzir/tenzir/pull/2887)

The bundled Zeek schema no longer includes the `_path` field included in Zeek JSON. Use `#type == "zeek.foo"` over `_path == "foo"` for querying data ingested using `vast import zeek-json`.

### Rename count, int, real, and addr to uint64, int64, double, and ip respectively

Jan 15, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2864](https://github.com/tenzir/tenzir/pull/2864)

The builtin types `count`, `int`, `real`, and `addr` were renamed to `uint64`, `int64`, `double`, and `ip` respectively. For backwards-compatibility, VAST still supports parsing the old type tokens in schema files.

### Move event distribution statistics to the catalog

Jan 12, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2852](https://github.com/tenzir/tenzir/pull/2852)

The per-schema event distribution moved from `index.statistics.layouts` to `catalog.schemas`, and additionally includes information about the import time range and the number of partitions VAST knows for the schema. The number of events per schema no longer includes events that are yet unpersisted.

### Change boolean literals to `true` and `false`

Jan 10, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2844](https://github.com/tenzir/tenzir/pull/2844)

Boolean literals in expressions have a new syntax: `true` and `false` replace the old representations `T` and `F`. For example, the query `suricata.alert.alerted == T` is no longer valid; use `suricata.alert.alerted == true` instead.

### Introduce a potpourri of smaller improvements

Jan 6, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2832](https://github.com/tenzir/tenzir/pull/2832)

VAST now ignores the previously deprecated options `vast.meta-index-fp-rate`, `vast.catalog-fp-rate`, `vast.transforms` and `vast.transform-triggers`. Similarly, setting `vast.store-backend` to `segment-store` now results in an error rather than a graceful fallback to the default store.

### Introduce a potpourri of smaller improvements

Jan 6, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2832](https://github.com/tenzir/tenzir/pull/2832)

Plugin names are now case-insensitive.

### PRs 2807-2848

Dec 23, 2022 · [@tobim](https://github.com/tobim) · [#2807](https://github.com/tenzir/tenzir/pull/2807)

Blocking imports now imply that ingested data gets persisted to disk before the the `vast import` process exits.

### Move taxonomy resolution to the catalog

Dec 20, 2022 · [@Dakostu](https://github.com/Dakostu) · [#2771](https://github.com/tenzir/tenzir/pull/2771)

`vast status` does not work anymore with an embedded node (i.e., spawned with the `-N` parameter).

### PRs 2693-2923

Dec 16, 2022 · [@patszt](https://github.com/patszt) · [#2693](https://github.com/tenzir/tenzir/pull/2693)

Building VAST now requires CAF 0.18.7. VAST supports setting advanced options for CAF directly in its configuration file under the `caf` section. If you were using any of these, compare them against the bundled `vast.yaml.example` file to see if you need to make any changes. The change has (mostly positive) [performance and stability implications](https://www.actor-framework.org/blog/2021-01/benchmarking-0.18/) throughout VAST, especially in high-load scenarios.

### Remove broker plugin

Dec 15, 2022 · [@patszt](https://github.com/patszt) · [#2796](https://github.com/tenzir/tenzir/pull/2796)

We removed the broker plugin that enabled direct Zeek 3.x log transfer to VAST. The plugin will return in the future rewritten for Zeek 5+.

### PRs 2778-2797-2798

Dec 14, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2778](https://github.com/tenzir/tenzir/pull/2778)

VAST no longer supports reading partitions created with VAST versions older than VAST v2.2. Since VAST v2.2, VAST continuously upgrades old partitions to the most recent internal format while running.

### Merge the type-registry into the catalog

Dec 10, 2022 · [@Dakostu](https://github.com/Dakostu) · [#2745](https://github.com/tenzir/tenzir/pull/2745)

`vast status` no longer shows type registry-related information. Instead, refer to `vast show` for detailed type metadata information.

### Add ‘pseudonymize’ pipeline operator

Dec 10, 2022 · [@Dakostu](https://github.com/Dakostu) · [#2719](https://github.com/tenzir/tenzir/pull/2719)

OpenSSL is now a required dependency.

### Remove the `#field` meta extractor

Dec 9, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2776](https://github.com/tenzir/tenzir/pull/2776)

The `#field` meta extractor no longer exists. Use `X != null` over `#field == "X"` to check for existence for the field `X`.

### PRs 2769-2873

Dec 9, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2769](https://github.com/tenzir/tenzir/pull/2769)

The match operator `~`, its negation `!~`, and the `pattern` type no longer exist. Use queries of the forms `lhs == /rhs/` and `lhs != /rhs/` instead for queries using regular expressions.

## 🐞 Bug Fixes

### Trigger new compaction runs immediately on error

Mar 10, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#3006](https://github.com/tenzir/tenzir/pull/3006)

Compaction now retries immediately on failure instead of waiting for the configured scan interval to expire again.

### Fix infinite recursion in the record algebra parser

Mar 1, 2023 · [@tobim](https://github.com/tobim) · [#2977](https://github.com/tenzir/tenzir/pull/2977)

VAST no longer crashes when it encounters an invalid type expression in a schema.

### Prevent query\_processor from hanging when there are no candidate partitions

Feb 8, 2023 · [@Dakostu](https://github.com/Dakostu) · [#2924](https://github.com/tenzir/tenzir/pull/2924)

The VAST client will now terminate properly when using the `count` command with a query which delivers zero results.

### Fix rounding issue when printing duration types

Feb 7, 2023 · [@patszt](https://github.com/patszt) · [#2906](https://github.com/tenzir/tenzir/pull/2906)

We fixed incorrect printing of human-readable durations in some edge cases. E.g., the value 1.999s was rendered as 1.1s instead of the expected 2.0s. This bug affected the JSON and CSV export formats, and all durations printed in log messages or the status command.

### Add a workaround to fix CAF OpenSSL options

Feb 2, 2023 · [@tobim](https://github.com/tobim) · [#2908](https://github.com/tenzir/tenzir/pull/2908)

Options passed in the `caf.openssl` section in the configuration file or as `VAST_CAF__OPENSSL__*` environment variables are no longer ignored.

### Remove the transformer actor

Feb 2, 2023 · [@Dakostu](https://github.com/Dakostu) · [#2896](https://github.com/tenzir/tenzir/pull/2896)

Pipelines that reduce the number of events do not prevent `vast export` processes that have a `max-events` limit from terminating any more.

### Align output of the Zeek TSV reader with schemas

Jan 28, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2887](https://github.com/tenzir/tenzir/pull/2887)

The Zeek TSV reader now respects the schema files in the bundled `zeek.schema` file, and produces data of the same schema as the Zeek JSON reader. E.g., instead of producing a top-level ip field `id.orig_h`, the reader now produces a top-level record field `id` that contains the ip field `orig_h`, effectively unflattening the data.

### Fix some shutdown issues in the web plugin

Jan 20, 2023 · [@lava](https://github.com/lava) · [#2860](https://github.com/tenzir/tenzir/pull/2860)

The web plugin now reacts correctly to CTRL-C by stopping itself.

### Run start commands asynchronously

Jan 17, 2023 · [@lava](https://github.com/lava) · [#2868](https://github.com/tenzir/tenzir/pull/2868)

The start commands specified with the `vast.start.commands` option are now run aynchronously. This means that commands that block indefinitely will no longer prevent execution of subsequent commands, and allow for correct signal handling.

### Fix linux PID file check on startup

Jan 15, 2023 · [@lava](https://github.com/lava) · [#2861](https://github.com/tenzir/tenzir/pull/2861)

VAST no longer ignores existing PID lock files on Linux.

### Introduce a potpourri of smaller improvements

Jan 6, 2023 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2832](https://github.com/tenzir/tenzir/pull/2832)

VAST now shuts down instantly when metrics are enabled instead of being held alive for up to the duration of the telemetry interval (10 seconds).

### Bump CAF to version 0.18.6

Dec 16, 2022 · [@patszt](https://github.com/patszt) · [#2693](https://github.com/tenzir/tenzir/pull/2693)

Attempting to connect with thousands of clients around the same time sometimes crashed the VAST server. This no longer occurs.

### Infer non-config types in extend and replace operators

Dec 10, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2768](https://github.com/tenzir/tenzir/pull/2768)

The `replace` and `extend` pipeline operators wrongly inferred IP address, subnet, pattern, and map values as strings. They are now inferred correctly. To force a value to be inferred as a string, wrap it inside double quotes.

[ Download on GitHub ](https://github.com/tenzir/tenzir/releases/tag/v3.0.0)

[Get the release artifacts and source code.](https://github.com/tenzir/tenzir/releases/tag/v3.0.0)