OpenCoreGovernance

Data Flows

View as Markdown

OpenCore separates source access, document processing, OpenSearch indexing, and query-time retrieval. Use these boundaries to decide where data is stored, which controls are applied, and when an external model provider can receive content.

Runtime boundaries

BoundaryData handledGovernance responsibility
Source systems and connectorsSource records and connector-specific access contextLimit connector credentials and source scope to the content approved for ingestion.
Processing workersFetched content, parsed text, chunks, embeddings, and optional enrichmentControl the configured parsing, embedding, vision, and contextual-retrieval features.
Relational metadataConnector configuration, indexing attempts, index settings, sharing references, and statusProtect administrative access and retain operational records according to deployment policy.
OpenSearchSearchable chunks, vectors, public or ACL metadata, filter metadata, and an optional tenant identifierProtect the search service, its network path, and the current index as application data.

OpenSearch documents do not contain connector credentials. They do contain searchable text and access-related metadata, so an OpenSearch snapshot must be governed as application content rather than as a disposable cache.

Indexing flow

1

Fetch approved source content

A connector reads records within its configured source and identity scope. Source-side permissions determine what the connector can fetch; OpenCore cannot infer a narrower source scope than the connector configuration supplies.

2

Parse and divide content

Workers extract supported content, split it into searchable chunks, and attach source, time, hierarchy, document-set, project, and sharing metadata when those values are available.

3

Run configured enrichment

OpenCore creates embeddings for semantic or hybrid retrieval. Contextual retrieval and image captioning can send relevant content to the selected model providers only when those optional features and models are configured.

4

Write the OpenSearch index

The current index receives chunk text, vectors, source links, filters, public or ACL markers, and other retrieval metadata. Indexing status is recorded separately from the searchable document.

Query flow

1

Authorize the request

The calling endpoint authenticates the requester and establishes its role, token scope, tenant scope, and resource context before retrieval.

2

Prepare retrieval

Keyword retrieval uses query text. Semantic and hybrid retrieval also embed the query with the current index settings.

3

Search the current index

OpenSearch evaluates the retrieval query against the current index and applies the filters supplied by the caller. Permission-aware retrieval combines public content with ACL entries that match the requester.

4

Return governed context

The retrieval layer returns filtered chunks to its caller. Chat or Agent workflows may then include those chunks in a model request according to the selected workflow and configured model provider.

OpenSearch applies ACL filtering only when the retrieval caller supplies an access-control scope. Endpoint authorization and tenant scoping remain required; search metadata is not a replacement for either control.

Query-time filters

FilterWhen it appliesBoundary
Public or ACL visibilityPermission-aware retrievalMatches public chunks or an ACL entry supplied for the requester.
TenantMulti-tenant deploymentsRestricts retrieval to the active tenant.
Source, tags, document sets, projects, or AgentsWhen the caller selects those scopesNarrows candidates to the requested knowledge boundary.
Time, hidden status, document, chunk, or hierarchyWhen the workflow supplies those valuesNarrows operational or content-level candidates.

The filters are combined as restrictions. Verify the caller’s filter construction as well as the OpenSearch mapping when auditing an access boundary.

Re-index and port lifecycle

Changing embedding or contextual-retrieval settings creates a future OpenSearch index. The current implementation ports stored chunks from the present index, recalculates the required enrichment or embeddings, and writes the result to the future index; it does not refetch every source record through its connector.

StrategyPromotion triggerQuery behavior
Re-index All Connectors Then SwitchEvery required port for active or paused, indexable connectors succeeds and deferred metadata work reaches zeroThe present index serves queries until promotion.
Re-index Active Connectors Then SwitchEvery required port for active connectors succeeds and deferred metadata work reaches zeroThe present index serves queries until promotion.
Switch Before Re-indexThe administrator explicitly selects immediate switch and the future index is createdThe future index becomes current immediately and is filled by the background port, so search can be incomplete during backfill.

Switch Before Re-index does not clear the OpenSearch corpus in the current backend. It promotes the future index and then backfills it through the port flow. Use this strategy only when temporary search gaps are acceptable, and do not start another model change until its backfill has finished.

Cancellation and concurrent changes

The port uses create-only writes so that a newer write already present in the destination is not overwritten by an older copied chunk. It also checks for deleted documents while copying, and metadata updates are applied to both indexes while a secondary index exists.

Canceling a pending change expires its future indexing work, stops its ports, marks that search setting as historical, and keeps or restores the prior current setting. Cancellation does not promise immediate deletion of every physical OpenSearch index.

Conditions and operations

ConditionEffectOperational check
OpenSearch-backed indexing is disabledConnector indexing and vector retrieval are unavailableConfirm the deployment mode before exposing Index Settings or connector workflows.
OpenSearch is unavailable or misconfiguredIndexing, migration, and retrieval can failValidate connectivity, authentication, TLS, capacity, and index health without exposing credentials.
Contextual retrieval or image processing is enabledSelected model providers can receive document content for enrichmentReview provider data handling and model configuration before enabling the feature.
Multi-tenant mode is enabledIndexed chunks and queries carry a tenant boundaryVerify tenant filters with representative accounts from different tenants.

Verify

  1. Index a representative document and confirm that its searchable chunks carry the expected source, visibility, and filter metadata.
  2. Query as an intended member and as an account outside the share; verify both the endpoint authorization and returned OpenSearch results.
  3. Start a non-immediate model change and confirm that queries continue to use the present index until all required ports and deferred metadata work complete.
  4. If testing an immediate switch, confirm that promotion occurs before backfill, that no clearing step is reported by the backend, and that result coverage grows as the port completes.
  5. Cancel a test migration and confirm that the previous current setting remains usable and no second migration starts while an immediate-switch backfill is pending.