S3

Amazon Simple Storage Service (S3) is an object storage service. Tenzir can treat it like a local filesystem to read and write files.

Use from_s3 to read files from S3. It supports glob patterns, automatic format detection, and IAM role assumption. For writing, use to_s3 with a print operator.

Configuration

See AWS Authentication for the available AWS authentication mechanisms.

Examples

Read from an S3 bucket

from_s3 "s3://my-bucket/path/to/file.json.gz"

Read all JSON files from an S3 bucket

from_s3 "s3://my-bucket/logs/**.json"

Write to an S3 bucket

from {foo: 42}
to_s3 "s3://my-bucket/path/to/file.json.gz" {
  write_json
  compress_gzip
}