Efficient Data Migration: Large-scale Object Store Transfer from AWS to GCP Without Downtime

Downtime is a non-starter for organizations operating at scale. Migration of petabyte-scale datasets between AWS S3 and Google Cloud Storage (GCS) is often required for cost control, leveraging Google’s AI/ML stack, regulatory targeting, or simply hedging against single-vendor risk. Technical teams, however, face a sequence of bottlenecks: throughput limits, cross-cloud auth, consistency management, and orchestrating cutovers.

Below: a tested method to migrate S3 buckets to GCS with zero unplanned downtime, based on field-validated patterns, not wishful documentation.

Problem Statement

You have production data—dozens to thousands of TB—in S3 buckets, with business applications still writing data throughout the migration. Both the raw move and the final cutover must guarantee integrity and immediate availability. Budget for bandwidth and cloud-native transfer costs is finite.

Risks & Constraints

• Migration must not impact S3-origin workloads.
• Throughput throttling by either provider can break plans, especially above 10Gbps.
• Policy: data must be encrypted, tracked, and deletions auditable.
• Compliance: US/EU residency, no unsanctioned cross-region copies.
• Monitoring: no black-box “fire and forget”—engineering ops requires visibility into transfer state.

Step 1: Inventory & Scope

Unsurprisingly, most failures trace to poor initial scoping. Aggregate a manifest of targeted buckets, record object counts, total size, and frequency of new/modified writes. Use aws s3 ls --summarize --human-readable --recursive for sizing.

Example inventory output:

$ aws s3 ls s3://critical-bucket --recursive | wc -l
1792312
$ aws s3 ls s3://critical-bucket --recursive --human-readable --summarize
Total Objects: 1792312
Total Size: 95.2 TiB

If a bucket exceeds tens of millions of objects, STS transfer jobs may need to be sharded by prefix.

Gotcha: Don’t ignore IAM. S3 ACLs may not translate directly to GCS permissions; map policies up front.

Step 2: Prepare Destination (GCS)

• Create GCS buckets with regional location set to match regulatory demands (us-central1, europe-west1, etc).
• Enable bucket versioning (gsutil versioning set on gs://your-bucket) if rollback capability is required.
• Assign minimal IAM necessary: roles/storage.objectAdmin for the migration principal.
• If switching encryption scheme, pre-create customer-managed keys (CMEK).

Note: Pre-flight a single test file to verify permissions and location as root—avoid troubleshooting during the live sync.

Step 3: Bulk Data Transmission – Storage Transfer Service (STS)

Google’s Storage Transfer Service is optimized for S3 → GCS (as of May 2024, v2 supports parallelism >15Gbps per transfer agent). It authenticates using AWS access keys and supports prefix filtering.

Example config:

gcloud transfer jobs create \
    --source-s3-bucket=my-source-bucket \
    --aws-access-key-id=AKIA... \
    --aws-secret-access-key=***** \
    --sink-gcs-bucket=my-dest-bucket \
    --source-agent-pool=projects/.../agentPools/my-fast-pool \
    --include-prefixes="prod/data/" \
    --description="S3 bulk to GCS migration" \
    --project=my-gcp-project

Monitor job status:

gcloud transfer operations list --filter='metadata.jobName:"my-job"'
gcloud transfer operations pause <OPERATION_NAME>

Known Issue: Some S3 objects with legacy ACLs or special chars may fail to transfer—STS logs failures, allow for re-inspection.

Step 4: Achieve Near Real-Time Consistency

While the bulk transfer proceeds, production writes still hit S3. Continuous delta sync is required.

Two main patterns in use:

1. Scheduled Sync (Rclone v1.63+ or AWS CLI + gsutil):

rclone sync s3:my-source-bucket gs://my-dest-bucket --progress --transfers=64 --s3-chunk-size=64M

Run as a systemd timer every 1-5 minutes. On buckets exceeding 10 million objects, sync by prefix to avoid memory overload.

2. Event-Driven Replication:

• S3 triggers Lambda on object creation;
• Lambda publishes S3 key to SNS topic;
• GCP Pub/Sub (via AWS SNS HTTP push endpoint) receives notification.
• Cloud Function or Dataflow pipeline fetches object, writes to GCS.

Trade-off: Event-driven sync offers lower lag but increases engineering complexity and failure points (e.g., SNS delivery retries).

Step 5: Cutover & Validation

At minimum object lag, time to swap production workloads.

Checklist:

• Block application writes or transition to dual-write mode.
• Trigger final Rclone or STS incremental delta.
• Using gsutil du -s and aws s3 ls, perform database-style count-and-size parity checks. Spot-sample key objects with md5sum. For critical workflows, compare ETag vs. GCS CRC32C hashes.
• Update DNS, endpoints, and configuration in the app stack.
• Monitor for the “file not found” or “PermissionDenied” in logs. Catch: error rates here generally pinpoints cutoff oversights.

Example: Mismatched object count after reconciliation

Source: 1,792,332 objects, 95.2TiB
Target: 1,792,325 objects, 95.2TiB
FAILED: 7 objects missing (reports in error log: STS_ERROR_3012)

For full auditability, retain S3 in read-only rather than delete immediately; schedule S3 bucket lifecycle policies for delayed removal.

Step 6: Optimization (Bandwidth/Cost/Security)

Side Note

Alternative designs, such as deploying a dedicated transfer VM pool in both clouds to saturate custom transfer logic (using multi-threaded AWS SDK and gsutil -m cp), do exist. They're operationally heavier but allow for edge-case logging and finer error handling.

Summary

Petabyte-scale S3-to-GCS migrations entail significant orchestration—inventory, bandwidth, authentication, near real-time sync, and cutover. Blindly relying on managed wizards is rarely sufficient at scale; layering native bulk copy with continuous sync and with failover validation is the pragmatic approach.

If the result must be audit-perfect and zero-downtime, expect complexity. But with careful sequencing and operational visibility, this migration problem is fully tractable.

Practical Tip:
Always test with a throwaway bucket containing non-critical data. Validate both object consistency and permission mapping—oddities creep in, especially with ACL-heavy S3 histories.

Not perfect, but:
No migration is “push-button.” Edge cases—API throttling, unexpected region latency, broken objects—are a constant. Build time for at least one “surprise” phase into your cutover plan.

References and Useful Links

• Google Cloud Storage Transfer Service docs
• AWS S3 CLI Reference
• rclone documentation
• gsutil command reference

If you’ve orchestrated a high-volume cross-cloud transfer and hit snags not listed here, add a note—the sharp edges aren’t always in the docs.