# Overview

This guide provides an overview of data collection in TQL. You’ll learn about the different approaches for ingesting data from various sources.

TQL provides two types of input operators:

* **`from_*` operators** like [`from_file`](https://preview.docs.tenzir.com/375/375/reference/operators/from_file.md) and

  [`from_http`](https://preview.docs.tenzir.com/375/375/reference/operators/from_http.md) read bytes and parse them using a [subpipeline](https://preview.docs.tenzir.com/375/375/reference/programs.md#parsing-subpipelines).

* **Direct event operators** like [`from_kafka`](https://preview.docs.tenzir.com/375/375/reference/operators/from_kafka.md) and [`accept_udp`](https://preview.docs.tenzir.com/375/375/reference/operators/accept_udp.md) produce structured events directly without an intermediate byte stream.

## Collection patterns

Different data sources require different collection approaches.

### Files and cloud storage

Read local files, watch directories for changes, or access cloud storage:

```tql
// Single file with automatic format detection
from_file "/var/log/app.json"


// Watch a directory for new files
from_file "/incoming/*.csv", watch=10s


// Cloud storage with glob patterns
from_file "s3://bucket/data/**/*.parquet"
```

See the [file reading guide](https://preview.docs.tenzir.com/375/375/guides/collecting/read-and-watch-files.md) for details.

### HTTP and APIs

Fetch data from web APIs with authentication, pagination, and retry handling:

```tql
from_http "https://api.example.com/events",
  headers={"Authorization": f"Bearer {secret("API_TOKEN")}"}
```

See the [HTTP and API guide](https://preview.docs.tenzir.com/375/375/guides/collecting/fetch-via-http-and-apis.md) for pagination patterns and advanced configurations.

### Message brokers

Subscribe to topics or queues from Apache Kafka, NATS, AMQP, Amazon SQS, and Google Cloud Pub/Sub:

```tql
from_kafka "security-events", offset="end"
```

For NATS JetStream, consume from a subject and parse the message field:

```tql
from_nats "alerts"
this = string(message).parse_json()
```

See the [message broker guide](https://preview.docs.tenzir.com/375/375/guides/collecting/read-from-message-brokers.md) for broker-specific configurations.

### Network data

Receive data over TCP or UDP sockets, or capture packets from network interfaces:

```tql
// UDP syslog receiver
accept_udp "0.0.0.0:514"


// TCP with TLS
accept_tcp "0.0.0.0:8443", tls={} {
  read_json
}
```

See the [network data guide](https://preview.docs.tenzir.com/375/375/guides/collecting/get-data-from-the-network.md) for socket configurations and packet capture.

## Sending data to destinations

For routing data to outputs, see the [Routing guides](https://preview.docs.tenzir.com/375/375/guides/routing/send-to-destinations.md), which cover destination operators, file output, load balancing, and pipeline connections.

## See also

* [Read and watch files](https://preview.docs.tenzir.com/375/375/guides/collecting/read-and-watch-files.md)
* [Fetch via HTTP and APIs](https://preview.docs.tenzir.com/375/375/guides/collecting/fetch-via-http-and-apis.md)
* [Read from message brokers](https://preview.docs.tenzir.com/375/375/guides/collecting/read-from-message-brokers.md)
* [Get data from the network](https://preview.docs.tenzir.com/375/375/guides/collecting/get-data-from-the-network.md)
* [Send to destinations](https://preview.docs.tenzir.com/375/375/guides/routing/send-to-destinations.md)

## Contents

- [Read-and-watch-files](https://preview.docs.tenzir.com/375/guides/collecting/read-and-watch-files.md)
- [Fetch-via-http-and-apis](https://preview.docs.tenzir.com/375/guides/collecting/fetch-via-http-and-apis.md)
- [Read-from-message-brokers](https://preview.docs.tenzir.com/375/guides/collecting/read-from-message-brokers.md)
- [Get-data-from-the-network](https://preview.docs.tenzir.com/375/guides/collecting/get-data-from-the-network.md)