# Tenzir Node v4.25.0 ## 馃殌 Features ### Custom quotes and doubled quote escaping Jan 7, 2025 路 [@IyeOnline](https://github.com/IyeOnline) 路 [#4837](https://github.com/tenzir/tenzir/pull/4837) The `read_csv`, `read_kv`, `read_ssv`, `read_tsv` and `read_xsv` operators now support custom quote characters. The `read_csv`, `read_ssv`, `read_tsv` and `read_xsv` operators support doubled quote escaping. The `read_csv`, `read_ssv`, `read_tsv` and `read_xsv` operators now accept multi-character strings as separators. The `list_sep` option for the `read_csv`, `read_ssv`, `read_tsv` and `read_xsv` operators can be set to an empty string, which will disable list parsing. The new `string.parse_leef()` function can be used to parse a string as a LEEF message. ### Implement `float(number|string)` Jan 7, 2025 路 [@raxyte](https://github.com/raxyte) 路 [#4882](https://github.com/tenzir/tenzir/pull/4882) Numbers and string expressions containing numbers can now be converted into `float` type values using the `float` function. ### Add user-defined operators to TQL2 Jan 6, 2025 路 [@jachris](https://github.com/jachris) 路 [#4884](https://github.com/tenzir/tenzir/pull/4884) User-defined operators can now be written and used in TQL2. To use TQL2, start your definition with the comment `// tql2`, or use the `--tql2` flag to opt into TQL2 as the default. ### Add `duration(string) -> duration` Dec 24, 2024 路 [@raxyte](https://github.com/raxyte) 路 [#4877](https://github.com/tenzir/tenzir/pull/4877) The new `duration` function now allows to parse expressions resulting in strings as duration values. ### Add Snowflake sink Dec 23, 2024 路 [@IyeOnline](https://github.com/IyeOnline) 路 [#4589](https://github.com/tenzir/tenzir/pull/4589) We have added a new `to_snowflake` sink operator, writing events into a [snowflake](https://www.snowflake.com/) table. ### Add `to_opensearch` and `to_elasticsearch` sink operators Dec 20, 2024 路 [@raxyte](https://github.com/raxyte) 路 [#4871](https://github.com/tenzir/tenzir/pull/4871) A new operator `to_opensearch` is now available for sending data to OpenSearch-compatible Bulk API providers including ElasticSearch. ### Implement TQL2 `from` and `to` Dec 18, 2024 路 [@IyeOnline](https://github.com/IyeOnline) 路 [#4805](https://github.com/tenzir/tenzir/pull/4805) We have added the `from` operator that allows you to easily onboard data from most sources. For example, you can now write `from "https://example.com/file.json.gz"` to automatically deduce the load operator, compression, and format. We have added the `to` operator that allows you to easily send data to most destinations. For example, you can now write `to "ftps://example.com/file.json.gz"` to automatically deduce the save operator, compression, and format. You can use the new `subnet(string)` function to parse strings as subnets. ### Enhance HTTP connector controls Dec 17, 2024 路 [@mavam](https://github.com/mavam) 路 [#4811](https://github.com/tenzir/tenzir/pull/4811) Several new options are now available for the `load_http` operator: `data`, `json`, `form`, `skip_peer_verification`, `skip_hostname_verification`, `chunked`, and `multipart`. The `skip_peer_verification` and `skip_hostname_verification` options are now also available for the `save_http` operator. ### Implement `context::erase` Dec 17, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4864](https://github.com/tenzir/tenzir/pull/4864) The `context::erase` operator allows you to selectively remove entries from contexts. ### Port `deduplicate` to TQL2 Dec 13, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4850](https://github.com/tenzir/tenzir/pull/4850) The `deduplicate` operator in TQL2 to help you remove events with a common key. The operator provides more flexibility than its TQL1 pendant by letting the common key use any expression, not just a field name. You can also control timeouts with finer granularity. ### Introduce a TQL2-only mode Dec 12, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4840](https://github.com/tenzir/tenzir/pull/4840) Start your Tenzir Node with `tenzir-node --tql2` or set the `TENZIR_TQL2=true` environment variable to enable TQL2-only mode for your node. In this mode, all pipelines will run as TQL2, with the old TQL1 pipelines only being available through the `legacy` operator. In Q1 2025, this option will be enabled by default, and later in 2025 the `legacy` operator and TQL1 support will be removed entirely. ### `save_email` cleanup Dec 11, 2024 路 [@raxyte](https://github.com/raxyte) 路 [#4848](https://github.com/tenzir/tenzir/pull/4848) The `save_email` now accepts a `tls` option to specify TLS usage when establishing the SMTP connection. ### Parse `x not in y` as `not x in y` Dec 10, 2024 路 [@raxyte](https://github.com/raxyte) 路 [#4844](https://github.com/tenzir/tenzir/pull/4844) TQL2 now allows writing `x not in y` as an equivalent to `not (x in y)` for better readability. ### Implement `ip in subnet` and `subnet in subnet` Dec 6, 2024 路 [@jachris](https://github.com/jachris) 路 [#4841](https://github.com/tenzir/tenzir/pull/4841) Whether an IP address is contained in a subnet can now be checked using expressions such as `1.2.3.4 in 1.2.0.0/16`. Similarly, to check whether a subnet is included in another subnet, use `1.2.0.0/16 in 1.0.0.0/8`. ## 馃敡 Changes ### Split `compress`/`decompress` into separate operators Jan 6, 2025 路 [@IyeOnline](https://github.com/IyeOnline) 路 [#4876](https://github.com/tenzir/tenzir/pull/4876) The `compress` and `decompress` operators have been deprecated in favor of separate operators for each compression algorithm. These new operators expose additional options, such as `compress_gzip level=10, format="deflate"`. ### Implement TQL2 `from` and `to` Dec 18, 2024 路 [@IyeOnline](https://github.com/IyeOnline) 路 [#4805](https://github.com/tenzir/tenzir/pull/4805) The `topic` argument for `load_kafka` and `save_kafka` is now a positional argument, instead of a named argument. The array version of `from` that allowed you to create multiple events has been removed. Instead, you can just pass multiple records to `from` now. ### Make the expression evaluator support heterogeneous results Dec 14, 2024 路 [@jachris](https://github.com/jachris) 路 [#4839](https://github.com/tenzir/tenzir/pull/4839) Functions can now return values of different types for the same input types. For example, `x.otherwise(y)` no longer requires that `x` has the same type as `y`. ## 馃悶 Bug Fixes ### Fix operator parenthesis continuation Jan 6, 2025 路 [@jachris](https://github.com/jachris) 路 [#4885](https://github.com/tenzir/tenzir/pull/4885) Operator invocations that directly use parenthesis but continue after the closing parenthesis are no longer rejected. For example, `where (x or y) and z` is now being parsed correctly. ### Fix handling of empty records in `write_parquet` Dec 20, 2024 路 [@dominiklohmann](https://github.com/dominiklohmann) 路 [#4874](https://github.com/tenzir/tenzir/pull/4874) `write_parquet` now gracefully handles nested empty records by replacing them with nulls. The Apache Parquet format does fundamentally not support empty nested records. ### Fix pipeline manager discarding parse-time warnings Dec 17, 2024 路 [@jachris](https://github.com/jachris) 路 [#4867](https://github.com/tenzir/tenzir/pull/4867) Warnings that happen very early during pipeline startup now correctly show up in the Tenzir Platform. ### Make the expression evaluator support heterogeneous results Dec 14, 2024 路 [@jachris](https://github.com/jachris) 路 [#4839](https://github.com/tenzir/tenzir/pull/4839) Metadata such as `@name` can now be set to a dynamically computed value that does not have to be a constant. For example, if the field `event_name` should be used as the event name, `@name = event_name` now correctly assigns the events their name instead of using the first value. ### Validate legacy expressions when splitting for predicate pushdown Dec 12, 2024 路 [@jachris](https://github.com/jachris) 路 [#4861](https://github.com/tenzir/tenzir/pull/4861) Pipelines that begin with `export | where` followed by an expression that does not depend on the incoming events, such as `export | where 1 == 1`, no longer cause an internal error. ### `save_email` cleanup Dec 11, 2024 路 [@raxyte](https://github.com/raxyte) 路 [#4848](https://github.com/tenzir/tenzir/pull/4848) The `endpoint` argument of the `save_email` operator was documented as optional but was not parsed as so. This has been fixed and the argument is now correctly optional. [ Download on GitHub ](https://github.com/tenzir/tenzir/releases/tag/v4.25.0) [Get the release artifacts and source code.](https://github.com/tenzir/tenzir/releases/tag/v4.25.0)