OpenCoreDocuments & KnowledgeConnectors

Amazon S3

View as Markdown

What it indexes

The S3 connector indexes objects from one bucket whose keys fall under the configured prefix. Leave the prefix blank for the whole bucket. A non-empty value without a trailing slash is normalized with /, so reports scopes listing to reports/ rather than keys such as reports-old.pdf.

Object behaviorIndexing effect
Folder markersKeys ending in / are skipped.
Text and documentsDownloaded objects pass through OpenCore’s common file extraction. An object with no extracted text can remain title-only; extraction failures are skipped.
TablesCSV, TSV, and XLSX objects are staged and indexed as tabular sections when the runtime file-staging callback is available.
ImagesImage objects are skipped unless image processing is enabled for the deployment.
SizeObjects larger than the blob storage threshold are skipped. The current default is 20 MiB unless the deployment overrides it.

Prerequisites

  • Choose one S3 bucket and, when needed, a key prefix ending at an approved folder boundary.
  • Provide an IAM principal that can list the bucket for that prefix and read the matching objects.
  • For IAM Role, allow the connector worker’s AWS identity to call STS and assume the configured Role ARN.
  • For Assume Role, run the connector worker with an instance or workload role available through the AWS credential chain.
  • Provide the bucket’s AWS Region for non-default partitions such as GovCloud; otherwise normal AWS region resolution is used.

Credentials

MethodOpenCore credential valuesNotes
Access Key and Secretauthentication_method: access_key, aws_access_key_id: <AWS_ACCESS_KEY_ID>, aws_secret_access_key: <AWS_SECRET_ACCESS_KEY>Uses a boto3 session created from the supplied key pair.
IAM Roleauthentication_method: iam_role, aws_role_arn: <AWS_ROLE_ARN>The worker uses its AWS environment to call STS AssumeRole, then refreshes temporary credentials.
Assume Roleauthentication_method: assume_roleDespite the UI label, this mode supplies no Role ARN; boto3 uses the worker’s existing instance or workload credentials directly.

These modes belong only to S3. Do not copy S3 authentication or Region fields into a Google Cloud Storage credential.

Configure in OpenCore

  1. In the Admin panel, open Connectors, select S3, and choose the credential mode that matches the worker’s AWS identity.
  2. Enter the exact Bucket Name. Do not include s3:// or an object path.
  3. Optionally enter Prefix. Use a key path such as approved/reports; OpenCore normalizes it to approved/reports/.
  4. Enter AWS Region when the bucket is in a non-default partition or when explicit regional resolution is required.
  5. Choose Private and assign OpenCore groups for restricted objects. Use Public only when every account may search every object in scope.
  6. Set refresh and cleanup options, create the connector, and run the first indexing attempt.

Permissions

The connector needs bucket-list access for the configured prefix and object-read access for every object to be indexed. Region detection also attempts a bucket-head request so that citations can point to the correct AWS console, and the iam_role mode needs STS AssumeRole access.

S3 bucket policies, object ACLs, IAM identities, and per-object authorization are not synchronized into OpenCore. Once an object is indexed, access is determined by the connector’s OpenCore Private groups or Public mode. A user also needs separate AWS console access to open an S3 citation link.

Verify

  1. Validate the connector and confirm it can list at least one object under the configured prefix.
  2. Run an indexing attempt and search for text from a recent object and a nested object.
  3. Confirm a key outside the normalized prefix is absent.
  4. Check that an oversized object and an image with image processing disabled are absent.
  5. Open a citation link with an AWS-authorized browser session and confirm the bucket, key, and Region are correct.
  6. Test every assigned OpenCore group because S3 permissions will not be re-evaluated at search time.

Troubleshooting

SymptomWhat to check
InvalidAccessKeyId or InvalidTokenVerify credentials and the configured Region/partition, especially for GovCloud.
AccessDenied during validationGrant list access on the bucket and ensure the bucket policy permits the selected principal.
SignatureDoesNotMatchCheck the key pair, Region, clock, and endpoint partition.
NoSuchBucket or 404Enter only the exact bucket name and confirm the worker is using the correct AWS partition.
Prefix returns no objectsRemember that reports becomes reports/; check key spelling, case, and the principal’s prefix condition.
Objects are missingCheck the 20 MiB default threshold, file extraction logs, image-processing setting, and object-read permission.
Citation opens the wrong AWS consoleSet the explicit Region and allow bucket-head detection; GovCloud uses its separate console domain.