# Tenzir Node v4.24.0 ## 🚀 Features ### Introduce a `zip` function for merging lists Dec 3, 2024 · [@dominiklohmann](https://github.com/dominiklohmann) · [#4803](https://github.com/tenzir/tenzir/pull/4803) The `zip` function merges two lists into a single list of a record with two fields `left` and `right`. For example, `zip([1, 2], [3, 4])` returns `[{left: 1, right: 3}, {left: 2, right: 4}]`. ### Port `unordered`, `local`, and `remote` to TQL2 Dec 3, 2024 · [@dominiklohmann](https://github.com/dominiklohmann) · [#4835](https://github.com/tenzir/tenzir/pull/4835) The `local` and `remote` operators allow for overriding the location of a pipeline. Local operators prefer running at a client `tenzir` process, and remote operators prefer running at a remote `tenzir-node` process. These operators are primarily intended for testing purposes. The `unordered` operator throws away the order of events in a pipeline. This causes some operators to run faster, e.g., `read_ndjson` is able to parse events out of order through this. This operator is primarily intended for testing purposes, as most of the time the ordering requirements are inferred from subsequent operators in the pipeline. ### Fix crash in `context::enrich` for heterogeneous enrichments Dec 2, 2024 · [@dominiklohmann](https://github.com/dominiklohmann) · [#4828](https://github.com/tenzir/tenzir/pull/4828) The `network` function returns the network address of a CIDR subnet. For example, `192.168.0.0/16.network()` returns `192.168.0.0`. ### Improve `to_splunk` TLS functionality Dec 2, 2024 · [@raxyte](https://github.com/raxyte) · [#4825](https://github.com/tenzir/tenzir/pull/4825) The `to_splunk` operator now supports the `cacert`, `certfile`, and `keyfile` options to provide certificates for the TLS connection. ### Implement `encode_hex()` and `decode_hex()` Dec 2, 2024 · [@raxyte](https://github.com/raxyte) · [#4815](https://github.com/tenzir/tenzir/pull/4815) The functions `encode_hex` and `decode_hex` transform strings and blobs to/from their hexadecimal byte representation. ### Implement `--limit` flag for the `chart` operator Nov 29, 2024 · [@IyeOnline](https://github.com/IyeOnline) · [#4757](https://github.com/tenzir/tenzir/pull/4757) The `--limit` option for the TQL1 `chart` operator controls the previously hardcoded upper limit on the number of events in a chart. The option defaults to 10,000 events. ### Allow aggregation functions to be called on lists Nov 28, 2024 · [@dominiklohmann](https://github.com/dominiklohmann) · [#4821](https://github.com/tenzir/tenzir/pull/4821) Aggregation functions now work on lists. For example, `[1, 2, 3].sum()` will return `6`, and `["foo", "bar", "baz"].map(x, x == "bar").any()` will return `true`. ### Implement `encode_base64` and `decode_base64` Nov 26, 2024 · [@raxyte](https://github.com/raxyte) · [#4806](https://github.com/tenzir/tenzir/pull/4806) The new functions `encode_base64` and `decode_base64` encode and decode blobs and strings as Base64. ### Implement `otherwise(, )` Nov 25, 2024 · [@raxyte](https://github.com/raxyte) · [#4794](https://github.com/tenzir/tenzir/pull/4794) The function `otherwise(primary:any, fallback:any)` provides a simple way to specify a `fallback` expression when the `primary` expression evaluates to `null`. ### Add `parse_time` and `format_time` methods Nov 21, 2024 · [@mavam](https://github.com/mavam) · [#4576](https://github.com/tenzir/tenzir/pull/4576) The new `parse_time` and `format_time` functions transform strings into timestamps and vice versa. ### Implement `split`, `split_regex`, and `join` Nov 21, 2024 · [@dominiklohmann](https://github.com/dominiklohmann) · [#4799](https://github.com/tenzir/tenzir/pull/4799) The `split` and `split_regex` functions split a string into a list of strings based on a delimiter or a regular expression, respectively. The `join` aggregation function concatenates a strings into a single string, optionally separated by a delimiter. ### Implement `append`, `prepend`, and `concatenate` Nov 21, 2024 · [@dominiklohmann](https://github.com/dominiklohmann) · [#4792](https://github.com/tenzir/tenzir/pull/4792) The new `append`, `prepend`, and `concatenate` functions add an element to the end of a list, to the front of a list, and merge two lists, respectively. `xs.append(y)` is equivalent to `[...xs, y]`, `xs.prepend(y)` is equivalent to `[y, ...xs]`, and `concatenate(xs, ys)` is equivalent to `[...xs, ..ys]`. ### Implement `x[y]` record indexing Nov 20, 2024 · [@raxyte](https://github.com/raxyte) · [#4795](https://github.com/tenzir/tenzir/pull/4795) Indexing records with string expressions is now supported. ### Implement `where` and `map` on lists Nov 20, 2024 · [@dominiklohmann](https://github.com/dominiklohmann) · [#4788](https://github.com/tenzir/tenzir/pull/4788) The `.map(, )` function replaces each value from `` with the value from ``. Within ``, the elements are available as ``. For example, to add 5 to all elements in the list `xs`, use `xs = xs.map(x, x + 5)`. The `.where(, )` removes all elements from `` for which the `` evaluates to `false`. Within ``, the elements are available as ``. For example, to remove all elements smaller than 3 from the list `xs`, use `xs = xs.where(x, x >= 3)`. ### Port Contexts to TQL2 Nov 14, 2024 · [@dominiklohmann](https://github.com/dominiklohmann) · [#4753](https://github.com/tenzir/tenzir/pull/4753) The contexts feature is now available in TQL2. It has undergone significant changes to make use of TQL2’s more powerful expressions. Contexts are shared between TQL1 and TQL2 pipelines. All operators are grouped in the `context` module, including the `enrich` and `show contexts` operators, which are now called `context::enrich` and `context::list`, respectively. To create a new context, use the `context::create_lookup_table`, `context::create_bloom_filter`, or `context::create_geoip` operators. Lookup table contexts now support separate create, write, and read timeouts via the `create_timeout`, `write_timeout`, and `read_timeout` options, respectively. The options are exclusive to contexts updated with TQL2’s `context::update` operator. ### PRs 4716-4807 Nov 11, 2024 · [@raxyte](https://github.com/raxyte) · [#4716](https://github.com/tenzir/tenzir/pull/4716) The following operators are now available in TQL2 for loading and saving: `load_amqp`, `save_amqp`, `load_ftp`, `save_ftp`, `load_nic`, `load_s3`, `save_s3`, `load_sqs`, `save_sqs`, `load_udp`, `save_udp`, `load_zmq`, `save_zmq`, `save_tcp` and `save_email`. The following new operators are available in TQL2 to convert event streams to byte streams in various formats: `write_csv`, `write_feather`, `write_json`, `write_lines`, `write_ndjson`, `write_parquet`, `write_pcap`, `write_ssv`, `write_tsv`, `write_xsv`, `write_yaml`, `write_zeek_tsv`. ### Support decapsulating SLL2 packets Nov 11, 2024 · [@mavam](https://github.com/mavam) · [#4744](https://github.com/tenzir/tenzir/pull/4744) The `decapsulate` function now handles SLL2 frames (Linux cooked capture encapsulation). ### Port `unroll` to TQL2 Nov 7, 2024 · [@dominiklohmann](https://github.com/dominiklohmann) · [#4736](https://github.com/tenzir/tenzir/pull/4736) The `unroll` operator is now available in TQL2. It takes a field of type list, and duplicates the surrounding event for every element of the list. ## 🔧 Changes ### Prepare small fixes for release Dec 3, 2024 · [@raxyte](https://github.com/raxyte) · [#4834](https://github.com/tenzir/tenzir/pull/4834) The new `string` function now replaces the `str` function. The older `str` name will be available as an alias for some time for compatibility but will be removed in a future release. ### Improve `to_splunk` TLS functionality Dec 2, 2024 · [@raxyte](https://github.com/raxyte) · [#4825](https://github.com/tenzir/tenzir/pull/4825) The `tls_no_verify` option of the `to_splunk` operator is now called `skip_peer_verification`. ### Align argument parser usage format with docs Nov 22, 2024 · [@jachris](https://github.com/jachris) · [#4740](https://github.com/tenzir/tenzir/pull/4740) The usage string that is reported when an operator or function is being used incorrectly now uses the same format as the documentation. ### Finish porting loaders, printers and savers Nov 19, 2024 · [@raxyte](https://github.com/raxyte) · [#4762](https://github.com/tenzir/tenzir/pull/4762) The option `ndjson` for `write_json` operator has been removed in favor of a new operator `write_ndjson`. ### Improve names for the `cache` operator’s timeout options Nov 13, 2024 · [@dominiklohmann](https://github.com/dominiklohmann) · [#4758](https://github.com/tenzir/tenzir/pull/4758) The `cache` operator’s `ttl` and `max_ttl` options are now called `read_timeout` and `write_timeout`, respectively. ### PRs 4741-4746 Nov 9, 2024 · [@jachris](https://github.com/jachris) · [#4741](https://github.com/tenzir/tenzir/pull/4741) The functions `ocsf_category_name`, `ocsf_category_uid`, `ocsf_class_name`, and `ocsf_class_uid` are now called `ocsf::category_name`, `ocsf::category_uid`, `ocsf::class_name`, and `ocsf::class_uid`, respectively. Similarly, the `package_add`, `package_remove`, `packages`, and `show pipelines` operators are now called `package::add`, `package::remove`, `package::list`, and `pipeline::list`, respectively. ### Stop URL-encoding pub/sub topics Nov 7, 2024 · [@dominiklohmann](https://github.com/dominiklohmann) · [#4738](https://github.com/tenzir/tenzir/pull/4738) The topics provided to the `publish` and `subscribe` operators now exactly match the `topic` field in the corresponding metrics. Using `publish` and `subscribe` without an explicitly provided topic now uses the topic `main` as opposed to an implementation-defined special name. ## 🐞 Bug Fixes ### Add timeout to multiline syslog Dec 3, 2024 · [@IyeOnline](https://github.com/IyeOnline) · [#4829](https://github.com/tenzir/tenzir/pull/4829) We fixed an oversight in the syslog parsers, which caused it to not yield an event until the next line came in. ### Add missing `co_yield`s in `save_http` Dec 3, 2024 · [@raxyte](https://github.com/raxyte) · [#4833](https://github.com/tenzir/tenzir/pull/4833) The TQL2 `save_http` operator had a bug causing it to fail to connect and get stuck in an infinite loop. This is now fixed and works as expected. ### Improve `to_splunk` TLS functionality Dec 2, 2024 · [@raxyte](https://github.com/raxyte) · [#4825](https://github.com/tenzir/tenzir/pull/4825) The `max_content_length` option for the `to_splunk` operator was named incorrectly in an earlier version to `send_timeout`. This has now been fixed. ### Fix ODR violation of `tenzir::socket` type Nov 26, 2024 · [@raxyte](https://github.com/raxyte) · [#4816](https://github.com/tenzir/tenzir/pull/4816) The TQL1 and TQL2 `sockets` operators no longer crash on specific builds. ### Fix `str` function quotes Nov 22, 2024 · [@jachris](https://github.com/jachris) · [#4809](https://github.com/tenzir/tenzir/pull/4809) The `str` function no longer adds extra quotes when given a string. For example, `str("") == "\"\""` was changed to `str("") == ""`. ### Ignore whole line when NDJSON parser fails Nov 21, 2024 · [@jachris](https://github.com/jachris) · [#4801](https://github.com/tenzir/tenzir/pull/4801) The `read_ndjson` operator no longer uses an error-prone mechanism to continue parsing an NDJSON line that contains an error. Instead, the entire line is skipped. ### Don’t allow manual erasing of contexts from packages Nov 14, 2024 · [@lava](https://github.com/lava) · [#4768](https://github.com/tenzir/tenzir/pull/4768) It is no longer possible to manually remove contexts that are installed as part of a package. ### Make `to_hive` a “local” operator Nov 14, 2024 · [@dominiklohmann](https://github.com/dominiklohmann) · [#4771](https://github.com/tenzir/tenzir/pull/4771) The `to_hive` operator now correctly writes files relative to the working directory of a `tenzir` client process instead of relative to the node. ### Port Contexts to TQL2 Nov 14, 2024 · [@dominiklohmann](https://github.com/dominiklohmann) · [#4753](https://github.com/tenzir/tenzir/pull/4753) The last metric emitted for each run of the `enrich` operator was incorrectly named `tenzir.enrich.metrics` instead of `tenzir.metrics.enrich`, causing it not to be available via `metrics enrich`. ### Prepend the field added by `enumerate` Nov 12, 2024 · [@dominiklohmann](https://github.com/dominiklohmann) · [#4756](https://github.com/tenzir/tenzir/pull/4756) The `enumerate` operator now correctly prepends the added index field instead of appending it. ### Port Loaders, Printers, Savers Nov 11, 2024 · [@raxyte](https://github.com/raxyte) · [#4716](https://github.com/tenzir/tenzir/pull/4716) The docs for the `sqs` connector now correctly reflect the default of `3s` for the `--poll-time` option. ### Introduce `{package,pipeline}::list` Nov 10, 2024 · [@dominiklohmann](https://github.com/dominiklohmann) · [#4746](https://github.com/tenzir/tenzir/pull/4746) `context inspect` crashed when used to inspect a context that was previously updated with `context update` with an input containing a field of type `enum`. This no longer happens. [ Download on GitHub ](https://github.com/tenzir/tenzir/releases/tag/v4.24.0) [Get the release artifacts and source code.](https://github.com/tenzir/tenzir/releases/tag/v4.24.0)