> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://developers.alephant.io/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://developers.alephant.io/_mcp/server. # Web > Configure rendered website indexing with explicit crawl, authentication, TLS, and access boundaries. ## What it indexes The Web connector turns rendered pages into OpenCore documents. Every run is a load-state crawl; start and end timestamps are ignored, so this connector does not provide incremental web polling. | Scrape method or content | Indexing behavior | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `recursive` | Starts at the Base URL and follows links found in rendered `` elements. A link stays in scope only when its normalized network authority (`netloc`, including an explicit port) matches after lowercasing and removing a leading `www.`, and its path is the configured path or a descendant. There is no Admin field for URL exclusions, include patterns, crawl depth, or a maximum page count. | | `single` | Renders only the configured URL and does not follow links from the page. For rendered HTML, a Playwright redirect changes the document ID and section link to the final URL. If the resource is detected as a PDF, the Requests download can follow a redirect but the document ID and section link retain the initially configured URL. | | `sitemap` | Fetches the configured sitemap URL, extracts its `` values, and renders those URLs. If the response has neither locations nor a `urlset`, a sitemap-discovery helper is attempted. The connector does not recursively follow links from the resulting pages or recursively parse child sitemap documents. | | HTML, PDF, and JavaScript | HTML is rendered with JavaScript enabled in headless Chromium, then cleaned into title and text. PDFs are detected by URL or `Content-Type` and extracted directly. **Scroll before scraping** optionally scrolls up to 20 times for lazy-loaded content. Web documents intentionally do not use `Last-Modified`. Rendered HTML pages are deduplicated by repeated title and content within one run; PDF documents do not enter that title-and-content deduplication set. | HTML cleanup removes deployment-wide ignored classes and elements. The defaults include the classes `sidebar,footer` and the elements `nav,footer,meta,script,style,symbol,aside`. Operators can replace those comma-separated lists with `WEB_CONNECTOR_IGNORED_CLASSES` and `WEB_CONNECTOR_IGNORED_ELEMENTS`; they are not per-connector fields. ## Prerequisites * Allow the OpenCore doc-fetching worker to resolve and reach every target URL, redirect, PDF, and sitemap location. * Install the Playwright Chromium runtime in the worker image. JavaScript rendering is always enabled; there is no Admin switch for static-only fetching. * Choose the narrowest Base URL path that contains the intended content. Recursive mode has no URL exclusion list, depth limit, or page-count limit. * Review the deployment SSRF protection setting. At the most restrictive level, the connector rejects URLs resolving to loopback, link-local, or private addresses; less restrictive settings change that boundary. * Use certificates trusted by the worker. Browser navigation is created with HTTPS errors ignored, but connectivity checks, sitemap requests, PDF downloads, and content-type probes use the HTTP client without a certificate override. The Admin form has no certificate-verification toggle. ## Credentials | Method | Configuration and boundary | | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Public website | No credential is created or selected in the Admin form. The connector sends fixed browser-like request headers; the form has no username, password, cookie, custom-header, bearer-token, or client-certificate field. | | Deployment-wide OAuth client credentials | If all three worker variables are set — `WEB_CONNECTOR_OAUTH_CLIENT_ID=`, `WEB_CONNECTOR_OAUTH_CLIENT_SECRET=`, and `WEB_CONNECTOR_OAUTH_TOKEN_URL=` — the shared browser context obtains a client-credentials token and sends it in the `Authorization` header. This is global worker configuration, not a credential scoped to one Web connector. | | Other protected sites | Per-connector custom headers and interactive login are not implemented by this Admin flow. Do not put secrets in the Base URL. Use a separately protected ingestion path when a global OAuth token is too broad. | Because HTTP probes and PDF requests use the connector's fixed default headers, the deployment-wide browser OAuth header does not establish that every non-browser fetch path is authenticated. Verify the exact target mix before relying on this mode. ## Configure in OpenCore 1. In the Admin panel, open **Connectors**, select **Web**, and enter a descriptive connector name. 2. Enter the exact **Base URL**. Include `https://` explicitly so the intended scheme is auditable, even though a missing scheme is normalized to HTTPS by the backend. 3. Select **recursive**, **single**, or **sitemap** according to the scope table above. For sitemap mode, enter the sitemap URL rather than the site home page unless discovery is intentional. 4. Enable **Scroll before scraping** only when required content appears after scrolling. This setting does not change single, recursive, or sitemap URL scope. 5. Select OpenCore **Private** groups for restricted content. Choose **Public** only when every OpenCore account may search every page the crawl can reach. 6. Set refresh and cleanup options, create the connector, and run an indexing attempt. Each refresh re-crawls the configured scope rather than applying an HTTP update window. ## Permissions The Web connector does not read or synchronize source-site users, sessions, groups, page ACLs, robots authorization, or per-page visibility. Web is not an **Auto Sync Permissions** source in the current Admin access selector. Search access is controlled only by the connector's OpenCore **Private** groups or **Public** mode. Any user with connector access can search every page emitted by the crawler, even when that user could not open the source page in a browser. A deployment-wide OAuth token also uses one shared source identity for all Web connectors on those workers. Use separate connectors and OpenCore Private groups for audiences that must not share content. Do not use this connector to crawl mixed public and restricted paths unless the entire emitted scope may be shared with the selected OpenCore audience. ## Verify 1. Run the connector and inspect the attempt status for connection, Playwright, TLS, HTTP, or sitemap errors. 2. Search for a distinctive title and sentence from the Base URL. Confirm navigation, footer, script, and other ignored content is absent as intended. 3. For recursive mode, test one descendant URL, one sibling path outside the Base URL, one URL on a different port, and one external-authority link. Only the descendant with the same normalized network authority should be indexed. 4. For single mode, confirm linked pages are absent. For sitemap mode, compare indexed document IDs with the sitemap locations and check that child sitemap URLs were not mistaken for content pages. 5. If scrolling is enabled, compare a known lazy-loaded sentence with a run that has scrolling disabled. 6. Test an HTML redirect and a PDF redirect separately. Rendered HTML redirects should use the final URL as document ID and link; PDF redirects should retain the initially configured URL. Also compare duplicate rendered HTML pages with duplicate PDFs because only rendered HTML uses title-and-content deduplication. 7. Test every assigned OpenCore group and a user outside those groups because the source site is not re-authorized at search time. ## Troubleshooting | Symptom | What to check | | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Recursive crawl is much larger than expected | Narrow the Base URL path. The current form has no URL exclusion, depth, or page-count field, and recursive scope follows same-site descendants discovered in rendered HTML. | | A same-site page is missing | Confirm its network authority (`netloc`, including an explicit port) matches after lowercasing and `www.` normalization, its path is under the Base URL path boundary, and a rendered anchor links to it. Links created only after interactions other than page load or optional scrolling may not be discovered. | | A redirected document shows an unexpected URL | Check the content path. Rendered HTML redirects replace the document ID and link with the final Playwright URL; PDF downloads may follow the redirect but retain the initially configured URL in the document. | | Sitemap creation fails or indexes XML | Provide a reachable sitemap containing direct page `` values. Child sitemap documents are not recursively expanded by the connector. | | Static HTML works but a PDF or probe returns 401/403 | The browser context can receive the deployment-wide OAuth header, while the connector's HTTP probes and PDF requests use fixed default headers. Use a supported ingestion path or redesign the source access boundary. | | A self-signed site fails although Chromium ignores HTTPS errors | The pre-crawl connectivity check and other HTTP-client paths still verify certificates. Install the issuing CA in the worker trust store; there is no per-connector bypass. | | Dynamic content is absent | Confirm Chromium is installed and scripts finish by the connector's load-state waits. Enable scrolling only when scroll-triggered loading is required; the connector has no configurable JavaScript delay. | | Users can search a page they cannot open at the source | This is expected because source ACLs are not synchronized. Restrict the connector with OpenCore Private groups or split the crawl by audience. | ## Related pages * [Connectors & Indexing](/opencore/knowledge/connectors) * [Index Settings](/opencore/knowledge/index-settings)