# Add pipelines

This guide shows you how to add deployable pipelines to your package. You’ll learn about pipeline frontmatter options and when to use pipelines versus operators.

## Create a pipeline

The `pipelines` directory contains fully deployable TQL pipelines. Unlike [user-defined operators (UDOs)](https://preview.docs.tenzir.com/375/375/guides/packages/add-operators.md), pipelines are complete units that must begin with an input operator and end with an output operator.

Pipelines typically use source operators to fetch and parse data, then call UDOs from the same package to transform the parsed events:

pipelines/fetch-and-normalize.tql

```tql
every 1h {
  from_http "https://api.example.com/events" {
    read_json
  }
}
acme::ocsf::map
ocsf::derive
ocsf::cast
publish "normalized-events"
```

When you install the package, the node automatically runs any enabled pipelines. The source operator handles fetching and parsing, while the package UDO handles cleanup, mapping, and other reusable transformation logic. Use enabled pipelines only when the package should take action as soon as it is installed. Ship optional operational workflows as disabled templates unless automatic execution is the package’s core behavior.

## Configure pipeline behavior

Configure pipelines using YAML frontmatter at the beginning of the TQL file. The following options are available:

### `name` and `description`

Add metadata to identify the pipeline in the UI:

pipelines/update-context.tql

```tql
---
name: Update Threat Intel
description: >
  Fetches threat intelligence hourly and updates the lookup table.
---


every 1h {
  from_http "https://feeds.example.com/iocs.json" {
    read_json
  }
}
context::update "threat-intel", key=indicator
```

### `restart-on-error`

Configure automatic restart behavior when the pipeline encounters an error. By default, pipelines stop running and show an error state.

pipelines/resilient-fetch.tql

```tql
---
restart-on-error: 5m  # Restart after 5 minutes on failure
---


every 1h {
  from_http "https://api.example.com/data"
}
publish "raw-data"
```

Options:

* Omit the option, or set it to `null` or `false` to disable automatic restarts
* Set it to `true` to enable restarts with a default delay of 1 minute
* Set it to a valid duration (for example, `5m`, `1h`) to enable restarts with a custom delay

### `disabled`

Set to `true` to disable the pipeline. Disabled pipelines don’t run when the package is installed, but users can enable them manually.

pipelines/notify-webhook.tql

```tql
---
name: Notify Webhook
description: >
  Sends selected security events to a webhook. Enable manually if needed.
disabled: true
---


subscribe "security-events"
where severity_id >= 3
to_http "https://hooks.example.com/security-events"
```

Ship disabled by default

For opt-in operation, add `disabled: true` to pipeline frontmatter. This ships the pipeline as a template that users must explicitly enable, combining the convenience of ready-made pipelines with predictable installation behavior.

### `unstoppable`

Set to `true` to make the pipeline run automatically and indefinitely. You cannot pause or stop unstoppable pipelines manually. If they complete, they end up in a failed state. If you enable `restart-on-error`, they restart after the specified duration.

pipelines/critical-router.tql

```tql
---
unstoppable: true
restart-on-error: 1m
---


from_kafka "critical-events"
publish "critical-events"
```

### Complete example

Combine multiple options:

pipelines/publish-as-ocsf.tql

```tql
---
name: Publish as OCSF
description: >
  Fetches threat intel hourly and publishes OCSF events to the `ocsf` topic.
disabled: true
restart-on-error: 5m
---


every 1h {
  acme::fetch
}
acme::ocsf::normalize
publish "ocsf"
```

## Operators versus pipelines

Understanding when to use operators versus pipelines helps you design packages that are both powerful and flexible.

| Aspect        | Operators                      | Pipelines                            |
| ------------- | ------------------------------ | ------------------------------------ |
| **Purpose**   | Reusable building blocks       | Complete workflows                   |
| **Execution** | Called explicitly by other TQL | Run automatically on install         |
| **Structure** | No input/output restrictions   | Must have input and output operators |
| **Testing**   | Test with sample data          | Test with fixtures or mocks          |

**Use operators when:**

* You want to provide reusable transformations
* The logic should be composable with other operations
* Users decide when and how to invoke the functionality

**Use pipelines when:**

* You need background tasks like periodic data fetching
* The workflow is complete and self-contained
* The package should take action upon installation

For flexible packages, consider shipping both: operators that provide reusable capabilities and disabled pipelines that demonstrate how to compose them.

## See also

* [Create a package](https://preview.docs.tenzir.com/375/375/guides/packages/create-a-package.md)
* [Add operators](https://preview.docs.tenzir.com/375/375/guides/packages/add-operators.md)
* [Add contexts](https://preview.docs.tenzir.com/375/375/guides/packages/add-contexts.md)
* [Write a package](https://preview.docs.tenzir.com/375/375/tutorials/write-a-package.md)
* [Packages](https://preview.docs.tenzir.com/375/375/explanations/packages.md)