from_http

Sends an HTTP/1.1 request and returns the response as events.

from_http url:string, [method=string, body=record|string|blob, encode=string,
          headers=record, error_field=field, paginate=string,
          paginate_delay=duration, connection_timeout=duration,
          max_retry_count=int, retry_delay=duration, tls=record]
          { … }

Description

The from_http operator issues an HTTP request and returns the response as events.

Format and Compression Inference

The from_http operator automatically infers the file format (such as JSON, CSV, Parquet, etc.) and compression type (such as gzip, zstd, etc.) directly from the URL’s file extension, just like the generic from operator. This makes it easier to load data from HTTP sources without manually specifying the format or decompression step.

If the format or compression cannot be determined from the URL, the operator will fall back to using the HTTP Content-Type and Content-Encoding response headers to determine how to parse and decompress the data.

If neither the URL nor the HTTP headers provide enough information, you can explicitly specify the decompression and parsing steps using a pipeline argument.

url: string

URL to connect to.

method = string (optional)

One of the following HTTP methods to use when using the client:

• get
• head
• post
• put
• del
• connect
• options
• trace

Defaults to get, or post if body is specified.

body = blob|record|string (optional)

Body to send with the HTTP request.

If the value is a record, then the body is encoded according to the encode option and an appropriate Content-Type is set for the request.

encode = string (optional)

Specifies how to encode record bodies. Supported values:

• json
• form

Defaults to json.

headers = record (optional)

Record of headers to send with the request.

paginate = record -> string | string (optional)

Controls automatic pagination of HTTP responses.

Lambda mode: A lambda expression to evaluate against the result of the request (optionally parsed by the given pipeline). If the expression evaluation is successful and non-null, the resulting string is used as the URL for a new GET request with the same headers.

Link mode: The string "link" to automatically follow pagination links in the HTTP Link response header per RFC 8288. The operator parses Link headers and follows the rel=next relation to fetch the next page. Pagination stops when the response no longer contains a rel=next link.

paginate_delay = duration (optional)

The duration to wait between consecutive pagination requests.

Defaults to 0s.

connection_timeout = duration (optional)

Timeout for the connection.

Defaults to 5s.

max_retry_count = int (optional)

The maximum times to retry a failed request. Every request has its own retry count.

Defaults to 0.

retry_delay = duration (optional)

The duration to wait between each retry.

Defaults to 1s.

error_field = field (optional)

Field to insert the response body for HTTP error responses (status codes not in the 2xx or 3xx range).

When set, any HTTP response with a status code outside the 200–399 range will have its body stored in this field as a blob. Otherwise, error responses are skipped and an error is emitted.

tls = record (optional)

TLS configuration. Provide an empty record (tls={}) to enable TLS with defaults or set fields to customize it.

{
  skip_peer_verification: bool, // skip certificate verification.
  cacert: string,               // CA bundle to verify peers.
  certfile: string,             // client certificate to present.
  keyfile: string,              // private key for the client certificate.
  min_version: string,          // minimum TLS version (`"1.0"`, `"1.1"`, `"1.2"`, "1.3"`).
  ciphers: string,              // OpenSSL cipher list string.
  client_ca: string,            // CA to validate client certificates.
  require_client_cert,          // require clients to present a certificate.
}

The client_ca and require_client_cert options are only applied for operators that accept incoming client connections, and otherwise ignored.

Any value not specified in the record will either be picked up from the configuration or if not configured will not be used by the operator.

See the Node TLS Setup guide for more details.

{ … } (optional)

A pipeline that receives the response body as bytes, allowing parsing per request. This is especially useful in scenarios where the response body can be parsed into multiple events.

Inside the pipeline, the $response variable is available as a record with the following fields:

Field	Type	Description
code	uint64	The HTTP status code of the response.
headers	record	The response headers.

If not provided, the operator will attempt to infer the parsing operator from the Content-Type header. Should this inference fail (e.g., unsupported or missing Content-Type), the operator raises an error.

Examples

Make a GET request

Make a request to urlscan.io to search for scans for tenzir.com and get the first result.

from_http "https://urlscan.io/api/v1/search?q=tenzir.com"
unroll results
head 1

{
  results: {
    submitter: { ... },
    task: { ... },
    stats: { ... },
    page: { ... },
    _id: "0196edb1-521e-761f-9d62-1ca4cfad5b30",
    _score: null,
    sort: [ "1747744570133", "\"0196edb1-521e-761f-9d62-1ca4cfad5b30\"" ],
    result: "https://urlscan.io/api/v1/result/0196edb1-521e-761f-9d62-1ca4cfad5b30/",
    screenshot: "https://urlscan.io/screenshots/0196edb1-521e-761f-9d62-1ca4cfad5b30.png",
  },
  total: 9,
  took: 296,
  has_more: false,
}

Send a POST request with JSON body

from_http "https://httpbin.org/post", body={key: "value"}, encode="json" {
  read_json
}

Access response metadata

Use the $response variable inside a parsing pipeline to access the HTTP response code and headers:

from_http "https://example.com/api", method="put" {
  read_json
  where $response.code == 200
  response = $response
}

Paginate via Link Headers

Use paginate="link" to automatically follow RFC 8288 Link headers with rel=next:

from_http "https://api.github.com/repos/tenzir/tenzir/issues?per_page=10",
  paginate="link"

Many APIs (such as GitHub, GitLab, and Jira) use the Link header for pagination. The operator extracts the rel=next URL from the header and continues fetching until no more pages are available.

Retry Failed Requests

Configure retries for failed requests:

from_http "https://api.example.com/data", max_retry_count=3, retry_delay=2s

This tries up to 3 times, waiting 2 seconds between each retry.

See Also

• accept_http
• http
• to_http
• serve_http
• serve
• Fetch via HTTP and APIs
• Enrich with threat intel
• Work with lookup tables
• HTTP(S)