OpenCoreDocuments & KnowledgeConnectors

Google Cloud Storage

View as Markdown

What it indexes

The Google Cloud Storage connector indexes objects from one GCS bucket through the storage.googleapis.com S3-compatible interoperability endpoint. Leave Path Prefix blank for the whole bucket. A non-empty value without a trailing slash is normalized with /, so reports lists reports/ rather than similarly named keys.

Object behaviorIndexing effect
Folder markersKeys ending in / are skipped.
Text and documentsDownloaded objects pass through OpenCore’s common file extraction. An object with no extracted text can remain title-only; extraction failures are skipped.
TablesCSV, TSV, and XLSX objects are staged and indexed as tabular sections when the runtime file-staging callback is available.
ImagesImage objects are skipped unless image processing is enabled for the deployment.
SizeObjects larger than the blob storage threshold are skipped. The current default is 20 MiB unless the deployment overrides it.

Prerequisites

  • Choose one GCS bucket and, when needed, a path prefix ending at an approved folder boundary.
  • Enable and create GCS interoperability HMAC credentials for a Google identity that can list the bucket and read matching objects.
  • Keep both parts of the HMAC key available at creation time; the secret cannot be recovered later from Google Cloud.
  • Allow connector workers to reach storage.googleapis.com over HTTPS.
  • Do not prepare a service account JSON key, Project ID, or Region field for this connector; the current credential and connector forms do not accept them.

Credentials

MethodOpenCore credential valuesNotes
GCS interoperability HMAC keyaccess_key_id: <GCS_ACCESS_KEY_ID>, secret_access_key: <GCS_SECRET_ACCESS_KEY>These are the GCS HMAC access ID and secret used by its XML/S3 interoperability API, not an AWS key pair or service account JSON file.

Google Cloud Storage and S3 share OpenCore’s blob indexing implementation, but they do not share a credential contract. GCS has no S3 authentication-method selector, Role ARN, or AWS Region field.

Configure in OpenCore

  1. In Google Cloud, create or activate the approved identity’s HMAC interoperability key and record both values securely.
  2. In the OpenCore Admin panel, open Connectors, select Google Cloud Storage, and create the GCS credential.
  3. Enter the exact Bucket Name without gs:// or an object path.
  4. Optionally enter Path Prefix. Use a key path such as approved/reports; OpenCore normalizes it to approved/reports/.
  5. Choose Private and assign OpenCore groups for restricted objects. Use Public only when every account may search every object in scope.
  6. Set refresh and cleanup options, create the connector, and run the first indexing attempt.

Permissions

The HMAC identity needs permission to list the configured bucket/prefix and read every matching object. OpenCore uses those credentials to ingest content, but it does not synchronize GCS IAM policies, bucket ACLs, or object ACLs.

After indexing, search access is determined by the connector’s OpenCore Private groups or Public mode. Any user with connector access can search every indexed object even if that user has no Google Cloud access. Opening a citation in the Google Cloud Console requires separate Google Cloud authorization.

Verify

  1. Validate the connector and confirm it can list at least one object under the configured prefix.
  2. Run an indexing attempt and search for text from a recent object and a nested object.
  3. Confirm a key outside the normalized prefix is absent.
  4. Check that an oversized object and an image with image processing disabled are absent.
  5. Open a citation with an authorized Google Cloud browser session and confirm it points to the expected bucket and object.
  6. Test every assigned OpenCore group because GCS permissions will not be re-evaluated at search time.

Troubleshooting

SymptomWhat to check
Credential is rejectedUse an active GCS interoperability HMAC access ID and secret, not a service account JSON key or AWS credentials.
AccessDenied during validationGrant the HMAC identity permission to list the bucket and verify that access is not blocked by the bucket IAM policy or ACL.
SignatureDoesNotMatchRe-enter the matching HMAC access ID and secret; rotate the HMAC key if the secret was lost.
NoSuchBucket or 404Enter only the exact bucket name and confirm it exists in the project accessible to the HMAC identity.
Prefix returns no objectsRemember that reports becomes reports/; check key spelling, case, and the identity’s prefix access.
Objects are missingCheck the 20 MiB default threshold, file extraction logs, image-processing setting, and object-read permission.
Users can search objects they cannot open in GCSThis is expected because source ACLs are not synchronized. Restrict the connector with OpenCore groups or split the source.