# Tenzir Node v2.0.0 ## 🚀 Features ### Print segment contents with lsvast Apr 29, 2022 · [@lava](https://github.com/lava) · [#2247](https://github.com/tenzir/tenzir/pull/2247) The `lsvast` tool now prints the whole store contents when given a store file as an argument. ### Clean up transform steps (and native plugins generally) Apr 20, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2228](https://github.com/tenzir/tenzir/pull/2228) The `replace` transform step now allows for setting values of complex types, e.g., lists or records. ### Implement support for transforms that apply to every type and use compaction for aging Apr 8, 2022 · [@lava](https://github.com/lava) · [#2186](https://github.com/tenzir/tenzir/pull/2186) VAST v1.0 deprecated the experimental aging feature. Given popular demand we’ve decided to un-deprecate it, and to actually implement it on top of the same building blocks the compaction mechanism uses. This means that it is now fully working and no longer considered experimental. ### Support environment variables as alternate config mechanism Apr 1, 2022 · [@mavam](https://github.com/mavam) · [#2162](https://github.com/tenzir/tenzir/pull/2162) VAST has now complete support for passing environment variables as alternate path to configuration files. Environment variables have *lower* precedence than CLI arguments and *higher* precedence than config files. Variable names of the form `VAST_FOO__BAR_BAZ` map to `vast.foo.bar-baz`, i.e., `__` is a record separator and `_` translates to `-`. This does not apply to the prefix `VAST_`, which is considered the application identifier. Only variables with non-empty values are considered. ### Backport bug fixes for a v1.1.1 release Mar 25, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2160](https://github.com/tenzir/tenzir/pull/2160) The disk monitor has new status entries `blacklist` and \`blacklist * size\` containing information about partitions failed to be erased. ### Allow fine-grained meta index configuration Mar 21, 2022 · [@lava](https://github.com/lava) · [#2065](https://github.com/tenzir/tenzir/pull/2065) The new `vast.index` section in the configuration supports adjusting the false-positive rate of first-stage lookups for individual fields, allowing users to optimize the time/space trade-off for expensive queries. ### Use dedicated partitions for each layout Mar 21, 2022 · [@tobim](https://github.com/tobim) · [#2096](https://github.com/tenzir/tenzir/pull/2096) VAST now creates one active partition per layout, rather than having a single active partition for all layouts. The new option `vast.active-partition-timeout` controls the time after which an active partition is flushed to disk. The timeout may hit before the partition size reaches `vast.max-partition-size`, allowing for an additional temporal control for data freshness. The active partition timeout defaults to 1 hour. ### Add a grand total event counter to the status output Mar 11, 2022 · [@6yozo](https://github.com/6yozo) · [#2133](https://github.com/tenzir/tenzir/pull/2133) The output of `vast status` now displays the total number of events stored under the key `index.statistics.events.total`. ## 🔧 Changes ### Remove the “catalog” and “catalog-bytes” keys from the index status Apr 27, 2022 · [@tobim](https://github.com/tobim) · [#2233](https://github.com/tenzir/tenzir/pull/2233) The `index` section in the status output no longer contains the `catalog` and `catalog-bytes` keys. The information is already present in the top-level `catalog` section. ### Clean up transform steps (and native plugins generally) Apr 20, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2228](https://github.com/tenzir/tenzir/pull/2228) Multiple transform steps now have new names: `select` is now called `where`, `delete` is now called `drop`, `project` is now called `put`, and `aggregate` is now called `summarize`. This breaking change is in preparation for an upcoming feature that improves the capability of VAST’s query language. The `layout-names` option of the `rename` transform step was renamed `schemas`. The step now additonally supports renaming `fields`. ### Eploit synergies when evaluating many queries at the same time Apr 8, 2022 · [@tobim](https://github.com/tobim) · [#2117](https://github.com/tenzir/tenzir/pull/2117) We revised the query scheduling logic to exploit synergies when multiple queries run at the same time. In that vein, we updated the related metrics with more accurate names to reflect the new mechanism. The new keys `scheduler.partition.materializations`, `scheduler.partition.scheduled`, and `scheduler.partition.lookups` provide periodic counts of partitions loaded from disk and scheduled for lookup, and the overall number of queries issued to partitions, respectively. The keys `query.workers.idle`, and `query.workers.busy` were renamed to `scheduler.partition.remaining-capacity`, and `scheduler.partition.current-lookups`. Finally, the key `scheduler.partition.pending` counts the number of currently pending partitions. It is still possible to opt-out of the new scheduling algorithm with the (deprecated) option `--use-legacy-query-scheduler`. ### Minimize the threadpool for client commands Apr 8, 2022 · [@tobim](https://github.com/tobim) · [#2193](https://github.com/tenzir/tenzir/pull/2193) Client commands such as `vast export` or `vast status` now create less threads at runtime, reducing the risk of hitting system resource limits. ### Fix CLI verbosity shorthands Apr 4, 2022 · [@mavam](https://github.com/mavam) · [#2178](https://github.com/tenzir/tenzir/pull/2178) The command line option `--verbosity` has the new name `--console-verbosity`. This synchronizes the CLI interface with the configuration file that solely understands the option `vast.console-verbosity`. ### Mark `experimental` encoding as `arrow.v2` Mar 31, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2159](https://github.com/tenzir/tenzir/pull/2159) VAST’s internal data model now completely preserves the nesting of the stored data when using the `arrow` encoding, and maps the pattern, address, subnet, and enumeration types onto Arrow extension types rather than using the underlying representation directly. This change enables use of the `export arrow` command without needing information about VAST’s type system. Transform steps that add or modify columns now transform the columns in-place rather than at the end, preserving the nesting structure of the original data. The deprecated `msgpack` encoding no longer exists. Data imported using the `msgpack` encoding can still be accessed, but new data will always use the `arrow` encoding. ### Remove the get subcommand Mar 6, 2022 · [@tobim](https://github.com/tobim) · [#2121](https://github.com/tenzir/tenzir/pull/2121) We removed the experimental `vast get` command. It relied on an internal unique event ID that was only exposed to the user in debug messages. This removal is a preparatory step towards a simplification of some of the internal workings of VAST. ### Bump the minimum version of Apache Arrow to 7.0 Mar 5, 2022 · [@tobim](https://github.com/tobim) · [#2122](https://github.com/tenzir/tenzir/pull/2122) VAST now requires Apache Arrow >= v7.0.0. ### Deploy VAST to AWS Lambda Mar 3, 2022 · [@rdettai](https://github.com/rdettai) · [#2108](https://github.com/tenzir/tenzir/pull/2108) VAST ships experimental Terraform scripts to deploy on AWS Lambda and Fargate. ### Rename `meta-index` to `catalog` Mar 3, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2128](https://github.com/tenzir/tenzir/pull/2128) The `meta-index` is now called the `catalog`. This affects multiple metrics and entries in the output of `vast status`, and the configuration option `vast.meta-index-fp-rate`, which is now called `vast.catalog-fp-rate`. ## 🐞 Bug Fixes ### Fix query pruning in the catalog May 9, 2022 · [@lava](https://github.com/lava) · [#2264](https://github.com/tenzir/tenzir/pull/2264) The query optimizer incorrectly transformed queries with conjunctions or disjunctions with several operands testing against the same string value, leading to missing result. This was rarely an issue in practice before the introduction of homogenous partitions with the v2.0 release. ### Ignore types unrelated to the configuration in the summarize plugin May 3, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2258](https://github.com/tenzir/tenzir/pull/2258) Transform steps removing all nested fields from a record leaving only empty nested records no longer cause VAST to crash. ### Lift selector field requirements for JSON import May 2, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2255](https://github.com/tenzir/tenzir/pull/2255) The JSON import no longer rejects non-string selector fields. Instead, it always uses the textual JSON representation as a selector. E.g., the JSON object `{id:1,...}` imported via `vast import json --selector=id:mymodule` now matches the schema named `mymodule.1` rather than erroring because the `id` field is not a string. ### Treat list options in env variables consistently Apr 26, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2236](https://github.com/tenzir/tenzir/pull/2236) Environment variables for options that specify lists now consistently use comma-separators and respect escaping with backslashes. ### Don’t send null pointers when erasing whole partitions Apr 19, 2022 · [@lava](https://github.com/lava) · [#2227](https://github.com/tenzir/tenzir/pull/2227) VAST no longer sometimes crashes when aging or compaction erase whole partitions. ### Stop accepting new queries after initiating shutdown Apr 13, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2215](https://github.com/tenzir/tenzir/pull/2215) VAST servers no longer accept queries after initiating shutdown. This fixes a potential infinite hang if new queries were coming in faster than VAST was able to process them. ### Add support for reals in CSV without dot Apr 5, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2184](https://github.com/tenzir/tenzir/pull/2184) The CSV parser no longer fails when encountering integers when floating point values were expected. ### Make man-page creation more robust Apr 5, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2190](https://github.com/tenzir/tenzir/pull/2190) The `vast(1)` man-page is no longer empty for VAST distributions with static binaries. ### Reduce the default log queue size for client commands Mar 31, 2022 · [@tobim](https://github.com/tobim) · [#2176](https://github.com/tenzir/tenzir/pull/2176) We optimized the queue size of the logger for commands other than `vast start`. Client commands now show a significant reduction in memory usage and startup time. ### Fix race condition with exporter timeouts Mar 30, 2022 · [@lava](https://github.com/lava) · [#2167](https://github.com/tenzir/tenzir/pull/2167) Some queries could get stuck when an importer would time out during the meta index lookup. This race condition no longer exists. ### Use the timestamp type for inferred event timestamp fields in the Zeek reader Mar 17, 2022 · [@tobim](https://github.com/tobim) · [#2155](https://github.com/tenzir/tenzir/pull/2155) The `import zeek` command now correctly marks the event timestamp using the `timestamp` type alias for all inferred schemas. ### Load stores lazily Mar 14, 2022 · [@dominiklohmann](https://github.com/dominiklohmann) · [#2146](https://github.com/tenzir/tenzir/pull/2146) The `count --estimate` erroneously materialized store files from disk, resulting in an unneeded performance penalty. VAST now answers approximate count queries by solely consulting the relevant index files. ### Correctly terminate the explore command Mar 6, 2022 · [@tobim](https://github.com/tobim) · [#2120](https://github.com/tenzir/tenzir/pull/2120) The `explore` command now properly terminates after the requested number of results are delivered. [ Download on GitHub ](https://github.com/tenzir/tenzir/releases/tag/v2.0.0) [Get the release artifacts and source code.](https://github.com/tenzir/tenzir/releases/tag/v2.0.0)