S3

Amazon Simple Storage Service (S3) is an object storage service. Tenzir can treat it like a local filesystem to read and write files.

Use from_s3 to read files from S3. It supports glob patterns, automatic format detection, and IAM role assumption. For writing, use save_s3 with a print operator.

Configuration

Follow the standard configuration instructions to authenticate with your AWS credentials.

Examples

Read from an S3 bucket

from_s3 "s3://my-bucket/path/to/file.json.gz"

Read all JSON files from an S3 bucket

from_s3 "s3://my-bucket/logs/**.json"

Write to an S3 bucket

from {foo: 42}
print_json
compress_gzip
save_s3 "s3://my-bucket/path/to/file.json.gz"