Connectors and sync
Which sources connect to a knowledge base automatically, how credentials and OAuth are stored, what a full load and a polling interval are, pause and resume, deletions in the source, SharePoint specifics and typical errors.
A connector fills a knowledge base from an external system without manual uploading: first it does a full load, then it picks up changes on a schedule. This article describes the catalog of connectors, their lifecycle and what to pay attention to when setting them up.
Source catalog
Connectors are created in the account settings, in the "Connectors" section, with the "New Connector" button. The owner and the administrators of the organization can create and change them.
| Group | Connectors |
|---|---|
| File storage | Google Drive, OneDrive, SharePoint (files and lists), SMB File Share (a network folder), Yandex Disk |
| Documentation and tasks | Confluence, Jira, Azure DevOps (repositories, wikis, work items, pull requests), GitHub and GitLab (repository code) |
| Mail and correspondence | Outlook, Email (IMAP), Microsoft Teams, Zulip, Bitrix24 Open Channels |
| Accounting systems | 1C:Enterprise (catalogs, documents and registers through OData) |
| Industry and external bases | BDU FSTEC (a vulnerability database), legis.md (legal acts of Moldova), instante.justice.md (court decisions of Moldova) |
| Data intake and processing | Bluedot (call recordings through a webhook), Generic (a JSON configuration and an intake token), MTR Normalization (a reference list from Excel) |
A connector is bound to one knowledge base. When you create it, you can create a new base ("Create Dataset", the name is filled in automatically) or leave the connector without a base. Some connectors write the content straight into the index and do not create files: on the page of such a base, the sync state is shown instead of the document list, and the content is checked on the "Retrieval testing" tab.
Credentials
The way to sign in depends on the source:
- token, or login with password: Confluence, Jira (an Atlassian API token or an OAuth 2.0 service account), GitHub and GitLab (a personal token; public repositories are available without it, but with a request rate limit), Azure DevOps, SMB, IMAP, Zulip, 1C;
- OAuth 2.0: Google Drive ("Sign in with Google", or Client ID, Client Secret and Refresh Token, or a service account JSON), Yandex Disk (an OAuth token);
- a Microsoft Entra application: OneDrive, SharePoint, Outlook, Teams. Use "Sign in with Microsoft" or enter Client ID, Client Secret, Tenant ID and Refresh Token manually. The application needs permissions for the task: sites for SharePoint, mail reading for Outlook, channel messages and files for Teams.
Secrets are stored in an encrypted store bound to the organization and are not returned to the interface; when editing, leave the secret field empty to keep the current value. The "Test Connection" button makes a trial request before saving. Create a separate read-only service account for each source.
Full load and schedule
After saving, the connector moves from the "New" state to "Syncing": a full load of all permitted content starts. Many sources have a "Backfill Since" field that limits the depth of history by a date. When the load finishes, the state becomes "Active", and the card shows "Last sync" and the counters of loaded items, skipped items and errors.
After that the connector works by polling with the period "Poll Interval (sec)"; the default value depends on the type. Zulip additionally supports a real-time events mode, and webhook connectors (Bluedot, Generic) do not poll for data but receive it by an intake token. Unchanged objects are skipped by version (eTag, modification date), so repeat passes are cheap. The "Backfill" button starts the traversal again without deleting what was already loaded.
On the base page, the status block shows the phase: "Up to date", "Syncing", "Initializing" (with steps, for example "Connecting", "Fetching pages", "Chunking and indexing"), "Paused", "Error", "New" or "Stuck". "Stuck" means that no worker holds the task; the "Clear and restart" button removes orphaned locks and continues from the saved position.
Pause, resume, deleting a connector
"Pause" stops polling, and the data in the base stays. "Resume" returns the connector to work from the saved position, not from the beginning. "Restart" is needed after changing credentials or filters. Deleting a connector deletes its secrets and its service index; the knowledge base with the documents is kept by default. The creation and deletion of connectors go to the audit log.
Deletions in the source
The behavior depends on the connector, and it is worth checking before launch:
- SMB has the "Remove missing files from the knowledge base" switch; it is off by default, and the cleanup is done only after a fully successful traversal.
- OneDrive gets a deletion flag from the change feed and removes the document from the base.
- SharePoint and 1C do not track deletions: a file or record deleted in the source stays in the base until it is deleted manually.
- Chats and Confluence have the "Handle deletions" setting.
If deletions are not tracked, assign someone responsible for reconciling the base with the source.
SharePoint specifics
In the content section you specify "Site URLs" like https://company.sharepoint.com/sites/my-site, "Document Libraries" (a list of names; empty means all libraries of the site), "Lists" (the names of SharePoint lists) and the "Include subsites" switch. Library files are processed as ordinary documents with the method set in the connector; files larger than "Max file size (MB)" are skipped and counted in the counter. File changes are read from the change feed of the site, while list items are turned into text documents with the title and the fields of the item and are traversed in full on every pass.
Errors to expect
| Message or state | Cause | What to do |
|---|---|---|
| "Connection failed" | Wrong credentials, an expired token, the application has no permissions | Repeat "Test Connection", renew the secret, check the permissions in the source |
| "Initialization error", the "Error" state after creation | No site, project or folder from the settings was found | Check the addresses and names, check the rights of the service account |
| The notification "Synchronization of connector ... failed" | A failure of the latest pass | Open "Last error" in the card; after fixing it, click "Resume" |
| "Stuck" | The worker was stopped during a pass | "Clear and restart" |
| The skipped counter grows | Files above the limit, or images without a recognition model | Raise the limit, set up a model or deliberately turn on "Skip images without vision model" |
Connectors and 1C
The 1C connector is arranged differently from the file sources. Its main task is to give assistants and agents tools for live reading of a base through OData: balances, turnovers, counterparty cards and arbitrary queries are executed at the moment of the question, not over a copy. In addition, it can index selected sets of entities into a knowledge base on a schedule. Preparing the publication, the 1C user and the analysis of errors are described in the article Connecting a 1C base through OData.
Changes that connectors notice on the next pass are available to automations as events; for how to react to them with an agent, see Agents on the canvas.
Ask Sema