from_amazon_sqs

Receives messages from an Amazon SQS queue.

from_amazon_sqs queue:str, [keep_messages=bool, batch_size=int, poll_time=duration,
                            visibility_timeout=duration, aws_region=str,
                            aws_iam=record]

Description

Amazon Simple Queue Service (Amazon SQS) is a fully managed message queueing service for decoupling distributed systems. The from_amazon_sqs operator reads messages from an SQS queue and emits one event per message.

The emitted events use the tenzir.sqs schema with these fields:

Field	Type	Description
message	string	The SQS message body.
message_id	string	The SQS message ID.
sent_time	time	The time when SQS sent the message.
first_receive_time	time	The first time SQS delivered the message.
receive_count	int	The approximate receive count.
sender_id	string	The sender ID reported by SQS.
message_group_id	string	The FIFO message group ID.
message_deduplication_id	string	The FIFO message deduplication ID.
sequence_number	string	The FIFO sequence number assigned by SQS.

All fields except message and message_id are optional because SQS only returns them when they are present on the message.

The operator uses long polling, which reduces empty responses when there are no messages available. By default, after the operator emits a message, it deletes the message from the queue.

The operator requires the following AWS permissions:

• sqs:GetQueueUrl (only when passing a queue name; not required for URLs)
• sqs:ReceiveMessage
• sqs:DeleteMessage (unless keep_messages=true)

queue: str

The name of the queue to receive messages from. You can optionally prefix the queue name with sqs://.

You can also pass a full queue URL (https://... or http://...) instead of a name. When a URL is given, the operator skips the GetQueueUrl lookup and talks to the URL’s host directly. Set aws_region when the queue is in a non-default AWS region so SigV4 signing matches.

poll_time = duration (optional)

The long polling timeout per request.

The value must be between 1s and 20s.

Defaults to 3s.

batch_size = int (optional)

The maximum number of messages to receive per SQS request.

The value must be between 1 and 10.

Defaults to 1.

keep_messages = bool (optional)

Whether to keep messages in the queue after receiving and emitting them.

Defaults to false.

Set this option to true to keep received messages in the queue:

from_amazon_sqs "my-queue", keep_messages=true

When keep_messages=true, SQS hides each received message until the queue’s visibility timeout expires, then makes the message visible again. This option only skips DeleteMessage; it doesn’t make downstream processing transactional.

visibility_timeout = duration (optional)

The receive-level visibility timeout for messages returned by SQS.

The value must be between 0s and 12h. When omitted, SQS uses the queue’s configured visibility timeout.

Use this option with keep_messages=true to control when messages become available for redelivery:

from_amazon_sqs "my-queue", keep_messages=true, visibility_timeout=30s

aws_region = str (optional)

The AWS region for resolving the queue URL and receiving messages.

If omitted, the operator uses the region from aws_iam when present. Otherwise, it uses the default AWS SDK region resolution.

aws_iam = record (optional)

Configures explicit AWS credentials or IAM role assumption. If not specified, the operator uses the AWS SDK's default credential chain.

{
  region: string,            // AWS region for API requests.
  access_key_id: string,     // AWS access key ID.
  secret_access_key: string, // AWS secret access key.
  session_token: string,     // session token for temporary credentials.
  assume_role: string,       // ARN of IAM role to assume.
  session_name: string,      // session name for role assumption.
  external_id: string,       // external ID for role assumption.
  web_identity: record,      // OIDC web identity token configuration.
}

See AWS Authentication for a description of every field, the default credential chain, web identity configuration, and local authentication with the AWS CLI.

Examples

Receive messages from a queue

from_amazon_sqs "sqs://tenzir"

Parse JSON messages

from_amazon_sqs "sqs://alerts", poll_time=5s, batch_size=10
this = message.parse_json()

Receive messages without deleting them

from_amazon_sqs "sqs://alerts",
  keep_messages=true,
  poll_time=5s,
  batch_size=10,
  visibility_timeout=30s
this = message.parse_json()

Use an explicit region

from_amazon_sqs "my-queue", aws_region="us-east-1"

Pass a queue URL directly

from_amazon_sqs "https://sqs.eu-west-1.amazonaws.com/123456789012/my-queue",
  aws_region="eu-west-1"

Use explicit credentials

from_amazon_sqs "my-queue", aws_iam={
  region: "us-east-1",
  access_key_id: secret("aws-key"),
  secret_access_key: secret("aws-secret")
}

Assume an IAM role

from_amazon_sqs "my-queue", aws_iam={
  region: "eu-west-1",
  assume_role: "arn:aws:iam::123456789012:role/my-sqs-role",
  session_name: "tenzir-session"
}

See Also

• to_amazon_sqs
• Read from message brokers
• SQS