> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://developers.alephant.io/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://developers.alephant.io/_mcp/server.

# Data Flows

> Understand how content moves through connector indexing, OpenSearch retrieval, and index migration.

OpenCore separates source access, document processing, OpenSearch indexing, and query-time retrieval. Use these boundaries to decide where data is stored, which controls are applied, and when an external model provider can receive content.

## Runtime boundaries

| Boundary                      | Data handled                                                                                           | Governance responsibility                                                                    |
| ----------------------------- | ------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- |
| Source systems and connectors | Source records and connector-specific access context                                                   | Limit connector credentials and source scope to the content approved for ingestion.          |
| Processing workers            | Fetched content, parsed text, chunks, embeddings, and optional enrichment                              | Control the configured parsing, embedding, vision, and contextual-retrieval features.        |
| Relational metadata           | Connector configuration, indexing attempts, index settings, sharing references, and status             | Protect administrative access and retain operational records according to deployment policy. |
| OpenSearch                    | Searchable chunks, vectors, public or ACL metadata, filter metadata, and an optional tenant identifier | Protect the search service, its network path, and the current index as application data.     |

OpenSearch documents do not contain connector credentials. They do contain searchable text and access-related metadata, so an OpenSearch snapshot must be governed as application content rather than as a disposable cache.

## Indexing flow

#### Fetch approved source content

A connector reads records within its configured source and identity scope. Source-side permissions determine what the connector can fetch; OpenCore cannot infer a narrower source scope than the connector configuration supplies.

#### Parse and divide content

Workers extract supported content, split it into searchable chunks, and attach source, time, hierarchy, document-set, project, and sharing metadata when those values are available.

#### Run configured enrichment

OpenCore creates embeddings for semantic or hybrid retrieval. Contextual retrieval and image captioning can send relevant content to the selected model providers only when those optional features and models are configured.

#### Write the OpenSearch index

The current index receives chunk text, vectors, source links, filters, public or ACL markers, and other retrieval metadata. Indexing status is recorded separately from the searchable document.

## Query flow

#### Authorize the request

The calling endpoint authenticates the requester and establishes its role, token scope, tenant scope, and resource context before retrieval.

#### Prepare retrieval

Keyword retrieval uses query text. Semantic and hybrid retrieval also embed the query with the current index settings.

#### Search the current index

OpenSearch evaluates the retrieval query against the current index and applies the filters supplied by the caller. Permission-aware retrieval combines public content with ACL entries that match the requester.

#### Return governed context

The retrieval layer returns filtered chunks to its caller. Chat or Agent workflows may then include those chunks in a model request according to the selected workflow and configured model provider.

OpenSearch applies ACL filtering only when the retrieval caller supplies an access-control scope. Endpoint authorization and tenant scoping remain required; search metadata is not a replacement for either control.

## Query-time filters

| Filter                                             | When it applies                         | Boundary                                                          |
| -------------------------------------------------- | --------------------------------------- | ----------------------------------------------------------------- |
| Public or ACL visibility                           | Permission-aware retrieval              | Matches public chunks or an ACL entry supplied for the requester. |
| Tenant                                             | Multi-tenant deployments                | Restricts retrieval to the active tenant.                         |
| Source, tags, document sets, projects, or Agents   | When the caller selects those scopes    | Narrows candidates to the requested knowledge boundary.           |
| Time, hidden status, document, chunk, or hierarchy | When the workflow supplies those values | Narrows operational or content-level candidates.                  |

The filters are combined as restrictions. Verify the caller's filter construction as well as the OpenSearch mapping when auditing an access boundary.

## Re-index and port lifecycle

Changing embedding or contextual-retrieval settings creates a future OpenSearch index. The current implementation ports stored chunks from the present index, recalculates the required enrichment or embeddings, and writes the result to the future index; it does not refetch every source record through its connector.

| Strategy                               | Promotion trigger                                                                                               | Query behavior                                                                                                                  |
| -------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| Re-index All Connectors Then Switch    | Every required port for active or paused, indexable connectors succeeds and deferred metadata work reaches zero | The present index serves queries until promotion.                                                                               |
| Re-index Active Connectors Then Switch | Every required port for active connectors succeeds and deferred metadata work reaches zero                      | The present index serves queries until promotion.                                                                               |
| Switch Before Re-index                 | The administrator explicitly selects immediate switch and the future index is created                           | The future index becomes current immediately and is filled by the background port, so search can be incomplete during backfill. |

**Switch Before Re-index** does not clear the OpenSearch corpus in the current backend. It promotes the future index and then backfills it through the port flow. Use this strategy only when temporary search gaps are acceptable, and do not start another model change until its backfill has finished.

## Cancellation and concurrent changes

The port uses create-only writes so that a newer write already present in the destination is not overwritten by an older copied chunk. It also checks for deleted documents while copying, and metadata updates are applied to both indexes while a secondary index exists.

Canceling a pending change expires its future indexing work, stops its ports, marks that search setting as historical, and keeps or restores the prior current setting. Cancellation does not promise immediate deletion of every physical OpenSearch index.

## Conditions and operations

| Condition                                           | Effect                                                               | Operational check                                                                                    |
| --------------------------------------------------- | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| OpenSearch-backed indexing is disabled              | Connector indexing and vector retrieval are unavailable              | Confirm the deployment mode before exposing Index Settings or connector workflows.                   |
| OpenSearch is unavailable or misconfigured          | Indexing, migration, and retrieval can fail                          | Validate connectivity, authentication, TLS, capacity, and index health without exposing credentials. |
| Contextual retrieval or image processing is enabled | Selected model providers can receive document content for enrichment | Review provider data handling and model configuration before enabling the feature.                   |
| Multi-tenant mode is enabled                        | Indexed chunks and queries carry a tenant boundary                   | Verify tenant filters with representative accounts from different tenants.                           |

## Verify

1. Index a representative document and confirm that its searchable chunks carry the expected source, visibility, and filter metadata.
2. Query as an intended member and as an account outside the share; verify both the endpoint authorization and returned OpenSearch results.
3. Start a non-immediate model change and confirm that queries continue to use the present index until all required ports and deferred metadata work complete.
4. If testing an immediate switch, confirm that promotion occurs before backfill, that no clearing step is reported by the backend, and that result coverage grows as the port completes.
5. Cancel a test migration and confirm that the previous current setting remains usable and no second migration starts while an immediate-switch backfill is pending.

## Related pages

* [Connectors & Indexing](/opencore/knowledge/connectors) explains source setup and connector status.
* [Index Settings](/opencore/knowledge/index-settings) explains embedding and contextual-retrieval controls.
* [Permissions Migration](/opencore/governance/permissions-migration) separates current authorization behavior from its future migration target.