Writing to S3 from AWS Lambda

Lambda and S3 fit together naturally: a function receives an event, produces a file, writes it to a bucket. The code is a single put_object call. What usually goes wrong is around it: IAM permissions, a trigger that invokes itself, payload size limits, or a function in a VPC that cannot reach S3. This post covers a working setup with Python and the AWS CLI, then the event-driven variant.

Create the bucket

aws s3api create-bucket --bucket my-app-output --region eu-central-1 \
  --create-bucket-configuration LocationConstraint=eu-central-1

New buckets block public access and encrypt objects with SSE-S3 by default, so nothing else is required for a private bucket. In us-east-1, omit --create-bucket-configuration.

IAM role for the function

The function needs a role that Lambda can assume, CloudWatch Logs permissions, and write access to exactly the prefix it uses.

aws iam create-role --role-name my-app-writer-role \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": { "Service": "lambda.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }]
  }'

aws iam attach-role-policy --role-name my-app-writer-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

aws iam put-role-policy --role-name my-app-writer-role --policy-name s3-write \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-app-output/events/*"
    }]
  }'

Avoid s3:* and AmazonS3FullAccess. If the bucket uses SSE-KMS with a customer managed key, also allow kms:GenerateDataKey on that key (and kms:Decrypt if the function reads objects back).

The function

# lambda_function.py
import json
import os
from datetime import datetime, timezone

import boto3

s3 = boto3.client("s3")
BUCKET = os.environ["BUCKET_NAME"]


def lambda_handler(event, context):
    now = datetime.now(timezone.utc)
    key = f"events/{now:%Y/%m/%d}/{context.aws_request_id}.json"
    s3.put_object(
        Bucket=BUCKET,
        Key=key,
        Body=json.dumps(event).encode("utf-8"),
        ContentType="application/json",
    )
    return {"bucket": BUCKET, "key": key}

Create the client outside the handler so warm invocations reuse it. The bucket name comes from an environment variable, not from the code. Using the request ID in the key makes every object unique, so retries do not overwrite each other. The Python runtimes include boto3, so there is nothing to package for this example.

Deploy and test

zip function.zip lambda_function.py

aws lambda create-function --function-name my-app-writer \
  --runtime python3.13 --handler lambda_function.lambda_handler \
  --role arn:aws:iam::123456789012:role/my-app-writer-role \
  --zip-file fileb://function.zip --timeout 30 \
  --environment 'Variables={BUCKET_NAME=my-app-output}'

aws lambda invoke --function-name my-app-writer \
  --cli-binary-format raw-in-base64-out \
  --payload '{"order_id": 42}' response.json

cat response.json
aws s3 ls s3://my-app-output/events/ --recursive

--cli-binary-format raw-in-base64-out lets AWS CLI v2 accept a plain JSON payload. If the role was created a few seconds earlier, create-function can fail because IAM has not propagated yet; wait and retry. For anything beyond an experiment, define the same resources in Terraform, AWS SAM or CDK.

Process uploads with an S3 trigger

A common pattern: files land in one place, the function transforms them and writes the result somewhere else.

# processor.py
import gzip
import os
import urllib.parse

import boto3

s3 = boto3.client("s3")
OUTPUT_BUCKET = os.environ["OUTPUT_BUCKET"]


def lambda_handler(event, context):
    for record in event["Records"]:
        bucket = record["s3"]["bucket"]["name"]
        key = urllib.parse.unquote_plus(record["s3"]["object"]["key"])
        body = s3.get_object(Bucket=bucket, Key=key)["Body"].read()
        s3.put_object(
            Bucket=OUTPUT_BUCKET,
            Key=f"processed/{key}.gz",
            Body=gzip.compress(body),
            ContentEncoding="gzip",
        )

Object keys arrive URL-encoded in the event (a space becomes +), hence unquote_plus. The role needs s3:GetObject on the source prefix and s3:PutObject on the output prefix.

Allow S3 to invoke the function, then add the notification:

aws lambda add-permission --function-name my-app-processor \
  --statement-id s3-invoke --action lambda:InvokeFunction \
  --principal s3.amazonaws.com \
  --source-arn arn:aws:s3:::my-app-uploads \
  --source-account 123456789012

aws s3api put-bucket-notification-configuration --bucket my-app-uploads \
  --notification-configuration '{
    "LambdaFunctionConfigurations": [{
      "LambdaFunctionArn": "arn:aws:lambda:eu-central-1:123456789012:function:my-app-processor",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": { "Key": { "FilterRules": [{ "Name": "prefix", "Value": "incoming/" }] } }
    }]
  }'

put-bucket-notification-configuration replaces the bucket's whole notification configuration, so include any existing entries.

Avoid recursive loops. If the function writes into the same bucket and prefix that triggers it, every output triggers a new invocation. Write to a different bucket, or at least to a prefix outside the trigger filter. Lambda's recursive loop detection, on by default, stops loops between Lambda and S3 after a number of iterations, but that is a safety net, not a design.

S3 delivers events at least once, so the same object can be processed twice. Make processing idempotent, for example by deriving the output key from the input key as above.

Large files and limits

• Payload size. Synchronous invocations accept at most 6 MB of request payload. Do not send files through Lambda. Let clients upload directly to S3 with a presigned URL generated by the function (s3.generate_presigned_url("put_object", Params={...}, ExpiresIn=900)).
• Memory. read() loads the whole object into memory. For large objects, stream with download_fileobj and upload_fileobj, which also switch to multipart uploads automatically.
• Temporary disk. /tmp is 512 MB by default and can be raised up to 10,240 MB with --ephemeral-storage Size=....
• Time. The maximum timeout is 15 minutes. Longer jobs belong in Step Functions, ECS or AWS Batch.

Lambda in a VPC

A function attached to private subnets has no internet access, and S3 is reached through the public endpoint. Add an S3 gateway VPC endpoint to the route tables of those subnets; it routes S3 traffic privately without a NAT gateway. Without it, S3 calls time out instead of failing with a clear error.

Troubleshooting

• AccessDenied on PutObject: the resource ARN in the policy does not match the key prefix, a bucket policy denies it, or KMS permissions are missing.
• Function times out on S3 calls: VPC without an S3 endpoint or NAT.
• Trigger does not fire: missing lambda:InvokeFunction permission for s3.amazonaws.com, or the prefix and suffix filters do not match.
• Logs: aws logs tail /aws/lambda/my-app-writer --follow.

Quick reference

aws iam put-role-policy ...   # s3:PutObject on arn:aws:s3:::BUCKET/prefix/*
aws lambda create-function --runtime python3.13 --handler lambda_function.lambda_handler ...
aws lambda invoke --function-name FN --cli-binary-format raw-in-base64-out --payload '{}' out.json
aws lambda add-permission --principal s3.amazonaws.com --source-arn arn:aws:s3:::BUCKET ...
aws s3api put-bucket-notification-configuration --bucket BUCKET --notification-configuration file://notify.json
aws logs tail /aws/lambda/FN --follow