Send to destinations

This guide shows you how to send data to various destinations using TQL output operators. You’ll learn about message destinations, data stores, file output patterns, and expression-based serialization.

Destination operators

TQL provides to_* operators for sending events to various destinations. Message-oriented operators accept expressions for flexible serialization, while data store operators write structured rows directly.

Message brokers

Send events to message brokers like Kafka, Amazon Kinesis Data Streams, and NATS.

Send to Kafka with automatic JSON formatting:

subscribe "security-events"
to_kafka "events"

Specify explicit serialization with the message parameter:

subscribe "logs"
to_kafka "events", message=this.print_json()

The message parameter accepts any expression that evaluates to a string or blob.

Send to NATS JetStream:

subscribe "security-events"
to_nats "alerts"

The NATS server must have a JetStream stream that captures the subject you publish to.

Send to Kinesis with the default NDJSON serialization:

subscribe "security-events"
to_amazon_kinesis "security-events"

Data stores

Send events to data stores like ClickHouse and Snowflake.

Send structured events to ClickHouse:

subscribe "security-events"
to_clickhouse table="alerts", primary=time, mode="create_append", tls=false

Write batches to Snowflake with bulk ingestion:

export
to_snowflake \
  account_identifier="org-account",
  user_name="tenzir_user",
  password=secret("SNOWFLAKE_PASSWORD"),
  database="SECURITY",
  schema="PUBLIC",
  table="EVENTS"

These operators preserve event structure instead of requiring a message expression.

Analytics platforms

Send data to platforms like Splunk, OpenSearch, and Elasticsearch.

Send to a Splunk HEC endpoint:

subscribe "logs"
to_splunk "https://splunk.example.com:8088",
  hec_token=secret("SPLUNK_HEC_TOKEN")

Send to OpenSearch with index routing:

subscribe "security"
to_opensearch "https://opensearch.example.com:9200",
  action="index",
  index="security-events"

Cloud services

Route events to cloud destinations like Amazon SQS and Google Cloud Pub/Sub.

Send to SQS:

subscribe "notifications"
to_amazon_sqs "sqs://notifications", message=this.print_json()

Send to Pub/Sub:

subscribe "events"
to_google_cloud_pubsub project_id="my-project",
  topic_id="events",
  message=this.print_json()

File output

For writing to files, use a to_* destination operator with a printing subpipeline. This pattern separates serialization from storage.

Write JSON to a local file:

subscribe "logs"
to_file "output.json" {
  write_json
}

Write compressed Parquet:

export
to_file "archive.parquet.zst" {
  write_parquet
}

Write JSON Lines to S3:

to_file "s3://bucket/logs/events.jsonl" {
  write_json
}

Send newline-delimited JSON over TCP:

to_tcp "collector.example.com:5044" { write_ndjson }

For protocols that expect a delimiter after every message, print the event to a string and use write_delimited for the byte-stream framing:

to_tcp "collector.example.com:12201" {
  write_delimited this.print_ndjson(strip_null_fields=true), "\x00"
}

Expression-based serialization for message destinations

Message-oriented destination operators use expressions for flexible message formatting:

Serialize the entire event

Use the default event serialization:

to_kafka "events"

Serialize as compact JSON without nulls:

to_kafka "events", message=this.print_json(include_nulls=false)

Serialize specific fields

Send only a specific field:

to_kafka "alerts", message=alert_message

Combine fields into a formatted string:

to_kafka "metrics", message=f"{host}: {metric_name}={value}"

Dynamic routing

Route events to different destinations based on content:

to_kafka f"events.{event_type}"