What is a knowledge base
What a knowledge base in Gleyk consists of: documents, chunks, an embedding model and an index; how to create a base, who sees it and how an assistant works with several bases.
A knowledge base (in the interface and the API it is also called a dataset) is the source in which an assistant looks for an answer. This article explains what a base consists of, how to create one and how it relates to assistants.
What a base consists of
| Component | What it is |
|---|---|
| Document | An uploaded file or a record that came from a connector. A document has a name, size, upload date, chunking method and processing status |
| Chunk | A part of a document that search finds and that the assistant cites in its answer. After processing, a document consists of tens or thousands of chunks |
| Embedding model | A model that converts each chunk into a numeric vector. Vectors are used to find chunks that are close in meaning to the question |
| Index | A store of chunks with vectors and a full-text index by words. Search combines both methods: by keywords and by meaning |
Document processing (parsing) does two things: it cuts the file into chunks taking its structure into account, and it writes the chunks into the index with vectors and keywords. Until a document is processed, the assistant does not see it.
The base page shows four counters: files, "Chunks", "Tokens" and "Size". A zero number of chunks for a document with real content means that processing did not succeed.
Creating a base
- On the top bar, open the "Dataset" section and click the button that creates a base.
- Enter a name. Within one organization the name must be unique.
- Open "Configuration". On the "General" tab you set the name, the image and the description; the description helps colleagues understand what the base is for and which documents are expected in it.
- On the "Document processing" tab, choose the embedding model and the PDF parser. The other processing parameters are hidden in the "Advanced settings" block and are described in the article Chunking methods.
- Click "Save" and go on to uploading: Uploading documents.
Warning
The embedding model is fixed once the first chunks appear. To switch to another model, you will have to delete all the documents in the base and process them again: vectors from different models cannot be compared with each other.
The owner and administrators of the organization, and the main administrator of the system, can create, rename, change the settings of and delete bases. Deleting a base deletes its documents and its index; files linked to the base from the "File Management" section stay there.
Who sees a base
Bases belong to an organization (tenant). All members of the organization see all of its bases, can search them, open documents and chunks, and upload and process documents. Resources of another organization are not visible: if the base you need is not in the list, first check the active organization in the account menu.
A base of another organization can be opened to you by the main administrator in "Read only" mode. Such a base is marked on its card; you can search it and connect it to assistants, but you cannot change anything. More in the article Knowledge base access and sharing.
How an assistant works with bases
An assistant is bound to one or several bases. When it gets a question, it searches for chunks in all connected bases at once, selects the best ones by the similarity threshold and passes them to the model together with the question. The answer shows the sources that were used: the document and the chunk, see Chat with sources.
A few rules when combining bases:
- All bases of one assistant must have the same embedding model, otherwise searching across them is impossible.
- If one base must take priority over another (for example, the current edition of regulations versus the archive), give it a higher PageRank in the processing settings.
- A base created as a set of tags is not connected to an assistant: it is used to label chunks of other bases.
- A base filled by a connector may contain no files: the content is written straight into the index. You can check what is inside on the "Retrieval testing" tab.
Before giving a base to an assistant, it is useful to run a control search: Checking search on a knowledge base.
What's next
- Which files you can upload and how to read the processing statuses: Uploading documents.
- How connectors fill a base without manual upload: Connectors and sync.
- How an answer from a knowledge base differs from an ordinary chat with a model: RAG vs general AI chat.
Ask Sema