Transferring Data Between AWS S3 and Google Cloud Storage

Moving buckets between AWS and Google Cloud is mostly a question of picking the right tool for the size and direction, giving it narrow permissions, and planning the final sync. Each cloud has a managed service that pulls data in from the other one, and both are better than a script once you are past a few hundred gigabytes.

Pick the tool

The managed services run inside the cloud, scale out on their own, retry failed objects and verify what they copied. Command-line tools send every byte through the machine that runs them, so run them on a VM in the destination cloud, not on a laptop.

S3 to Cloud Storage with Storage Transfer Service

Storage Transfer Service pulls objects from S3 without any agent. It needs read access to the source bucket: s3:ListBucket on the bucket and s3:GetObject on its objects (plus s3:DeleteObject only if you ask it to delete sources after transfer).

Prefer a federated IAM role over access keys. First get the subject ID of your project's transfer service account:

curl -s -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  https://storagetransfer.googleapis.com/v1/googleServiceAccounts/my-project

Create an IAM role in AWS that trusts Google as a web identity provider for that subject only:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "accounts.google.com" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": { "StringEquals": { "accounts.google.com:sub": "SUBJECT_ID" } }
  }]
}

Attach the read policy to the role, put its ARN in a credentials file and create the job:

echo '{"roleArn": "arn:aws:iam::123456789012:role/gcs-transfer-reader"}' > aws-role.json

gcloud transfer jobs create s3://my-source-bucket gs://my-dest-bucket \
  --source-creds-file=aws-role.json \
  --overwrite-when=different \
  --schedule-repeats-every=1d

gcloud transfer jobs monitor JOB_NAME

With access keys instead of a role the file contains accessKeyId and secretAccessKey. Use --include-prefixes to copy part of a bucket. --delete-from=destination-if-unique makes the destination an exact mirror; leave it off until you are sure nothing else writes there.

Storage Transfer Service can also route S3 transfers over a Google-managed private network. AWS then does not bill internet egress for the transfer, Google charges a per-GiB fee instead, and only a subset of AWS regions is supported. Compare both options for your volume.

Smaller copies with gcloud storage or rclone

gcloud storage reads S3 directly when it finds AWS credentials in ~/.aws/credentials:

gcloud storage rsync s3://my-source-bucket gs://my-dest-bucket --recursive --dry-run
gcloud storage rsync s3://my-source-bucket gs://my-dest-bucket --recursive

rsync only copies new and changed objects, so running it again is cheap. Add --delete-unmatched-destination-objects for a mirror. The same command works in the other direction with gs:// as source and s3:// as destination.

rclone does the same with more tuning knobs:

# ~/.config/rclone/rclone.conf
[s3]
type = s3
provider = AWS
env_auth = true
region = eu-central-1

[gcs]
type = google cloud storage
env_auth = true
bucket_policy_only = true

rclone sync s3:my-source-bucket gcs:my-dest-bucket --transfers 32 --dry-run
rclone sync s3:my-source-bucket gcs:my-dest-bucket --transfers 32
rclone check s3:my-source-bucket gcs:my-dest-bucket

Cloud Storage to S3 with AWS DataSync

DataSync reads Cloud Storage through its S3-compatible XML API. In Enhanced mode, transfers between Cloud Storage and S3 need no agent.

1. Create a service account with roles/storage.objectViewer on the source bucket and an HMAC key for it:

   gcloud storage hmac create gcs-reader@my-project.iam.gserviceaccount.com
   

2. Create the source location (no --agent-arns for an agentless transfer):

   aws datasync create-location-object-storage \
     --server-hostname storage.googleapis.com \
     --server-protocol HTTPS --server-port 443 \
     --bucket-name my-gcs-bucket \
     --access-key HMAC_ACCESS_ID --secret-key HMAC_SECRET
   

3. Create the destination with aws datasync create-location-s3 and an IAM role DataSync can assume to write to the bucket.
4. Create the task with --task-mode ENHANCED. Turn off copying object tags: the Cloud Storage XML API does not support them and the task can fail.
5. Run it with aws datasync start-task-execution, and run it again for each incremental pass.

DataSync can also store the HMAC secret in AWS Secrets Manager instead of taking it on the command line. If the transfer has to stay on private connectivity, you need an agent.

Incremental sync and cutover

Large buckets are never copied in one go during a maintenance window. The pattern:

1. Initial bulk copy days or weeks ahead.
2. Repeated incremental runs (scheduled job, rerun task, or rsync) that copy only new and changed objects.
3. At cutover, stop writers, run one last incremental pass, point applications at the new bucket.

If writers cannot stop, have the application write to both buckets for a while, or read from the new bucket and fall back to the old one until the last pass is done.

Validate the result

Compare object counts and total size on both sides:

aws s3 ls s3://my-source-bucket --recursive --summarize | tail -2
gcloud storage du -s gs://my-dest-bucket

Do not compare S3 ETags with Cloud Storage MD5 hashes: objects uploaded to S3 in multiple parts have ETags that are not MD5 sums. Storage Transfer Service and DataSync verify each object during transfer, and rclone check compares whatever hashes both sides support.

What does not carry over

• Bucket policies, ACLs and IAM: recreate them in the target model.
• Lifecycle rules, event notifications, replication and CORS settings.
• Object versions: by default only live versions are copied.
• Object tags: Cloud Storage has no object tags.
• Storage classes: set the target class on the job or the bucket, they do not map automatically.

Standard metadata such as Content-Type and Cache-Control comes across. Spot-check custom metadata on a few objects.

Cost notes

The source cloud bills data leaving it, plus request charges on both sides. Many small objects cost more in requests than a few large ones. Both AWS and Google waive egress for customers leaving their platform entirely, after an application to their support team.

Quick reference

# S3 to Cloud Storage, managed and scheduled
gcloud transfer jobs create s3://SRC gs://DST --source-creds-file=aws-role.json --schedule-repeats-every=1d

# Either direction, ad hoc
gcloud storage rsync s3://SRC gs://DST --recursive
gcloud storage rsync gs://SRC s3://DST --recursive

# Cloud Storage to S3, managed
aws datasync create-task --source-location-arn SRC_ARN --destination-location-arn DST_ARN --task-mode ENHANCED
aws datasync start-task-execution --task-arn TASK_ARN