Uploading Files to Google Cloud Storage: CLI, Code, Signed URLs and SFTP

Cloud Storage holds objects in buckets. There are no real folders, a name like reports/2026/q1.pdf is just a key with slashes. This post covers creating a bucket, the ways to get files into it, and what to change when files are large, numerous, come from a browser or arrive over FTP.

Create a bucket

gcloud storage buckets create gs://my-app-uploads \
  --location=europe-west1 \
  --uniform-bucket-level-access \
  --public-access-prevention

Bucket names are global, so pick something unique. Uniform bucket-level access means permissions come only from IAM, not per-object ACLs. Public access prevention blocks accidental public sharing.

Grant the least role that does the job:

• roles/storage.objectCreator: upload only, cannot read, overwrite or delete.
• roles/storage.objectViewer: read and list.
• roles/storage.objectUser or roles/storage.objectAdmin: full object access.

gcloud storage buckets add-iam-policy-binding gs://my-app-uploads \
  --member=serviceAccount:uploader@my-project.iam.gserviceaccount.com \
  --role=roles/storage.objectCreator

Upload from the console

In the Google Cloud console open Cloud Storage > Buckets, select the bucket and use Upload files or Upload folder, or drag files onto the page. Fine for a handful of files. For anything repeatable use the CLI.

Upload with gcloud storage

gcloud storage is the current CLI. gsutil still works but is legacy and slower.

gcloud storage cp report.pdf gs://my-app-uploads/reports/
gcloud storage cp -r ./site gs://my-app-uploads/site
gcloud storage cp "*.log" gs://my-app-uploads/logs/
gcloud storage ls gs://my-app-uploads/reports/

To keep a local directory in sync, copying only new and changed files:

gcloud storage rsync ./data gs://my-app-uploads/data --recursive --dry-run
gcloud storage rsync ./data gs://my-app-uploads/data --recursive

--delete-unmatched-destination-objects also removes objects that no longer exist locally. Use --dry-run first, that flag deletes data.

You can also stream from a pipe, which avoids a temporary file:

tar czf - /var/www | gcloud storage cp - gs://my-app-uploads/backups/www.tar.gz

Large files

Two mechanisms make large uploads reliable and fast, and gcloud storage uses both automatically.

Resumable uploads. The file is sent in chunks within one upload session. If the connection drops, the upload continues from the last confirmed chunk instead of starting over.

Parallel composite uploads. Files above the threshold (150 MiB by default) are split into up to 32 parts, uploaded in parallel and composed into one object. gcloud storage only does this when the object is in the Standard class and the bucket has no retention policy or default object holds. The resulting object has a CRC32C checksum but no MD5 hash, which matters if a downstream tool expects MD5. To turn it off:

gcloud config set storage/parallel_composite_upload_enabled False

Many small files

Per-request latency, not bandwidth, limits uploads of thousands of small files. gcloud storage cp -r and rsync already run in parallel. In Python, use the transfer manager instead of a loop:

from google.cloud.storage import Client, transfer_manager

bucket = Client().bucket("my-app-uploads")
files = ["a.json", "b.json", "c.json"]
results = transfer_manager.upload_many_from_filenames(
    bucket, files, source_directory="./out", blob_name_prefix="out/", max_workers=8
)
for name, result in zip(files, results):
    if isinstance(result, Exception):
        print(f"failed {name}: {result}")

If thousands of tiny files are only ever read together, upload one archive instead.

Upload from code

Python client (pip install google-cloud-storage), using Application Default Credentials (gcloud auth application-default login locally, the attached service account on Compute Engine, GKE or Cloud Run):

from google.cloud import storage

bucket = storage.Client().bucket("my-app-uploads")
blob = bucket.blob("reports/report.pdf")
blob.upload_from_filename("report.pdf", content_type="application/pdf", if_generation_match=0)

The client switches to resumable uploads for larger files on its own. if_generation_match=0 makes the upload fail if the object already exists, which protects against silent overwrites. Avoid downloading service account key files for this: attached service accounts and Workload Identity Federation do the same job without a long-lived secret.

Integrity

gcloud storage and the client libraries compute checksums and compare them with what Cloud Storage stored, so a corrupted upload fails instead of succeeding quietly. To check an object yourself:

gcloud storage hash report.pdf
gcloud storage objects describe gs://my-app-uploads/reports/report.pdf --format="value(crc32c_hash,md5_hash)"

Browser and client uploads with signed URLs

Do not proxy user uploads through your backend and do not hand out credentials. Generate a short-lived signed URL on the server and let the client PUT the file directly:

gcloud storage sign-url gs://my-app-uploads/incoming/photo.jpg \
  --http-verb=PUT --duration=15m \
  --headers=content-type=image/jpeg \
  --impersonate-service-account=signer@my-project.iam.gserviceaccount.com

In Python the same is blob.generate_signed_url(version="v4", expiration=timedelta(minutes=15), method="PUT", content_type="image/jpeg"). For uploads from a browser, set a CORS policy on the bucket that allows PUT from your origin. Validate size and type after upload, for example with a function triggered on object creation.

FTP and SFTP into Cloud Storage

Cloud Storage has no built-in FTP or SFTP endpoint. Common options:

SFTP server with periodic sync. Run OpenSSH on a small Compute Engine VM, let partners upload to a local directory, and push to the bucket from cron:

*/5 * * * * gcloud storage rsync /srv/sftp/incoming gs://my-app-uploads/incoming --recursive

Simple and robust. Files land in the bucket a few minutes later.

SFTP server on a Cloud Storage FUSE mount. Mount the bucket with gcsfuse and point a chrooted SFTP user at it:

sudo mkdir -p /srv/sftp/partner/upload
sudo gcsfuse --implicit-dirs --uid=$(id -u partner) --gid=$(id -g partner) \
  -o allow_other my-app-uploads /srv/sftp/partner/upload

The chroot directory (/srv/sftp/partner) must be owned by root and not writable by the user, so the bucket goes in a subdirectory. allow_other needs user_allow_other in /etc/fuse.conf. Files appear in the bucket as soon as the client closes them. FUSE is not a full POSIX file system: no file locking and renames are copy plus delete, which some clients use for partial uploads.

Pulling from an existing FTP server. rclone talks to FTP, SFTP and Cloud Storage directly:

rclone copy legacy-ftp:/outgoing gcs:my-app-uploads/incoming --transfers 8

Prefer SFTP over plain FTP: FTP sends passwords in clear text.

Large migrations

For tens of terabytes from on-premises storage, use Storage Transfer Service with agents installed near the data. It copies in parallel, resumes, and runs incremental passes. When the network is the limit, Transfer Appliance is a rack device you fill on-site and ship to Google. For data in other clouds see Transferring Data Between AWS S3 and Google Cloud Storage.

Storage classes and lifecycle

Colder classes are cheaper to store and more expensive to read, and deleting early is billed as if the minimum duration had passed. Set a class per object with --storage-class on cp, enable Autoclass on the bucket to let Google move objects by access pattern, or move them with lifecycle rules:

{
  "rule": [
    { "action": { "type": "SetStorageClass", "storageClass": "NEARLINE" }, "condition": { "age": 30 } },
    { "action": { "type": "Delete" }, "condition": { "age": 365 } }
  ]
}

gcloud storage buckets update gs://my-app-uploads --lifecycle-file=lifecycle.json

Quick reference

gcloud storage buckets create gs://BUCKET --location=REGION --uniform-bucket-level-access
gcloud storage cp FILE gs://BUCKET/path/
gcloud storage cp -r DIR gs://BUCKET/path
gcloud storage rsync DIR gs://BUCKET/path --recursive
gcloud storage sign-url gs://BUCKET/key --http-verb=PUT --duration=15m
gcloud storage buckets update gs://BUCKET --lifecycle-file=lifecycle.json