How to Access a Private GCS Bucket using a Load Balancer

Follow these steps to create a private Google Cloud Storage (GCS) bucket and configure access through a load balancer.

Create a Private GCS Bucket

1. Create a GCS Bucket: Make sure it’s private by selecting Enforce public access prevention on this bucket and choose Uniform for Access control.
2. Upload Your Content: Add your files to the bucket, for example image.jpg.

Create a Service Account

1. Navigate to IAM & Admin -> Service Accounts.
2. Create a Service Account: Add a name for the service account (referred to as PRIVATE_GCS_SA) and leave other settings as default.

Generate HMAC for the Service Account

1. Go to Cloud Storage -> Settings.
2. Switch to the Interoperability Tab.
3. Create an Access Key: Under “Access keys for service accounts,” click + Create a key for a Service Account.
4. Select the Service Account: Choose the previously created service account (PRIVATE_GCS_SA) and click Create Key.
5. Save the Access Key and Secret: Note the “Access key” (HMAC_ACCESS_KEY) and “Secret” (HMAC_SECRET) for later use.

Add Service Account Permissions to the Bucket

1. Go to Cloud Storage -> [BUCKET_NAME].
2. Switch to the PERMISSIONS Tab.
3. Grant Access: Click + GRANT ACCESS and type the service account email (PRIVATE_GCS_SA). Assign the “Storage Legacy Object Reader” role and click Save.

Create Network Endpoint Groups

1. Navigate to Compute Engine -> Network Endpoint Groups.
2. Create a Network Endpoint Group: Name it like ”private-gcs-neg”
3. Configure the Endpoint Group:
   • Type: Select “Internet NEG (Global, Regional).”
   • Default Port: 443.
   • Add Through: Choose “Fully qualified domain name and port.”
   • Domain Name: Input “[BUCKET_NAME].storage.googleapis.com.”
4. Create the Endpoint Group.

Create an HTTP(S) Load Balancer

1. Go to Network Services -> Load Balancing.
2. Create a Load Balancer: Click + Create Load Balancer.
3. Configure the Load Balancer:
   • Type: Select Application Load Balancer (HTTP/HTTPS).
   • Frontend Configuration:
     • Protocol: HTTPS (includes HTTP/2 and HTTP/3).
     • IP Address: Static IP address (or create a new one).
     • Certificate: Select or create a new one.
     • HTTP to HTTPS Redirect: Enable it.
   • Backend Configuration:
     • Create a Backend Service: Name it.
     • Backend Type: Select Internet network endpoint group.
     • Protocol: HTTPS.
     • Backend Selection: Choose the previously created Network Endpoint Group.
     • Cloud CDN: Ensure Cloud CDN is enabled.
     • Security: Select None for Cloud Armor backend security policy.
     • Custom Request Headers: Add a header with ”host” as the name and ”[BUCKET_NAME].storage.googleapis.com” as the value.
4. Create the Load Balancer: Set the name and finalize the creation.

Configure Private Origin Authentication

1. Open Cloud Shell or gcloud CLI.
2. Export Backend Configuration: Use the command:

   1
   gcloud compute backend-services export --global [Backend Service Name] \
   2
       --destination=private-gcs.yaml \
   3
       --project [PROJECT_ID]

3. Edit the YAML File: Add the following to the securitySettings section:

   1
   securitySettings:
   2
       awsV4Authentication:
   3
           accessKeyId: [HMAC_ACCESS_KEY]
   4
           accessKey: [HMAC_SECRET]
   5
           originRegion: [REGION]

   Replace [HMAC_ACCESS_KEY], [HMAC_SECRET], and [REGION] with the appropriate values.
4. Import the Configuration: Use the command:

   1
   gcloud compute backend-services import --global [Backend Service Name] \
   2
       --source=private-gcs.yaml \
   3
       --project [PROJECT_ID]

Testing

Run the following cURL command

1
curl -X GET -I https://[LOAD_BALANCER_IP]/image.jpg

IF success, it will return 200 similar like this

1
HTTP/1.1 200 OK
2
x-goog-generation: 1721373786382023
3
x-goog-metageneration: 1
4
x-goog-stored-content-encoding: identity
5
x-goog-stored-content-length: 1456
6
x-amz-meta-x-goog-reserved-source-generation: 1721283240953192
7
x-goog-hash: crc32c=Bbj32A==
8
x-goog-hash: md5=Gz8lupMp0rjzxJSMOWfbhQ==
9
x-amz-checksum-crc32c: Bbj32A==
10
x-goog-storage-class: STANDARD
11
accept-ranges: bytes
12
Content-Length: 1456
13
x-guploader-uploadid: ACJd0NqW9GOPmKlURsNWyGpDyxgA34g-BB9psFcQFai8XdB52HuwBbDEiO4B3ZJPOB91_ROPluE
14
server: UploadServer
15
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
16
via: 1.1 google
17
Date: Sat, 20 Jul 2024 06:37:12 GMT
18
Cache-Control: public,max-age=3600
19
Age: 17
20
Last-Modified: Fri, 19 Jul 2024 07:23:06 GMT
21
ETag: "1b3f25ba9329d2b8f3c4948c3967db85"
22
Content-Type: image/png
23
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

Reference

• https://cloud.google.com/cdn/docs/configure-private-origin-authentication