Checking search on a knowledge base
The "Retrieval testing" tab: similarity threshold, keyword and vector weight, Top-K, reranker, how to read the scores of found chunks, typical causes of poor search and the check to run before connecting a base to an assistant.
Search testing shows which chunks a base will return for a question, before an assistant is connected to it. If the right chunk is found here, look for the problem with an answer in the settings of the assistant and the model; if it is not found, look in the document processing.
Where and how to run it
- Open the knowledge base and go to the "Retrieval testing" tab. The documents must have the "SUCCESS" status.
- In the "Test text" field, enter a question whose correct source you know in advance.
- For the first run, do not change the parameters and click "Testing".
- In the "Test Results" block, check the first chunks and their scores. The "View" button opens the source document at the right place.
Repeat this for four types of questions: an exact question in the words of the document, a question with synonyms, a question about a table and a question whose answer is not in the base. The last one must not return a confident-looking but unrelated chunk above the threshold.
Parameters
| Parameter | What it does |
|---|---|
| Similarity threshold | Chunks whose hybrid score is below the threshold are dropped. The default is 0.2, that is 20 points on the results scale |
| Keyword similarity weight | The share of word similarity in the hybrid score; the remainder goes to vector similarity. The weights add up to 1 |
| Top-K | How many chunks are passed to the reranking model |
| Rerank model | Optional. If selected, the vector part of the score is replaced by the reranker score. It noticeably increases the response time |
| Use knowledge graph | Adds entities, relationships and community reports from the built graph to the search. It requires a model call and lengthens the answer |
| Cross-language search | One or several languages into which the model translates the question before the search. Needed when the documents and the questions are in different languages |
The search is hybrid: keyword match and vector closeness are computed separately and added up with weights. Without a reranker, the hybrid score equals "Term similarity" multiplied by the keyword weight plus "Vector similarity" multiplied by the remaining weight. For example, with weights 0.7 and 0.3, the scores 25.17 and 36.49 give 28.57. Chunks created by the knowledge graph are scored by vector similarity only.
Warning
The parameters you change on the tab are not saved anywhere. The selected values of the threshold, weights and reranker must be transferred to the settings of the assistant or of the agent's search component.
How to read the results
For each found chunk, "Hybrid similarity", "Term similarity" and "Vector similarity" are shown. Typical pictures and what they mean:
- High term similarity and low vector similarity: the words matched, but the meaning did not. This is often a chunk with the right terms from another context, for example from a table of contents.
- Low term similarity and high vector similarity: the question is worded differently. Such a chunk is useful, and the keyword weight can be lowered.
- The right chunk is there, but below the first three: refine the wording, or add keywords and questions to the chunk on the chunks page.
- The right chunk is not there at all: the cause is in the processing, see the table below.
Check that the numbers, dates and names in a chunk match the original: distortions at this stage mean a recognition problem, not a search problem.
Typical causes and fixes
| Symptom | Probable cause | What to do |
|---|---|---|
| Chunks start and break off in the middle of a phrase, a table is torn apart | Wrong chunking method | Change the method (for example, "Laws" for regulations, "Table" for registers) and process again, see Chunking methods |
| A chunk is found, but it does not contain the whole answer | The chunk size is too small | Increase "Recommended chunk size" or turn on "Structure-aware chunking" |
| The scores are low for all results, the chunks are long and heterogeneous | The chunk size is too large | Reduce the size, check the delimiter |
| The documents are in one language, the questions in another | No cross-language search | Turn on "Cross-language search" with the needed languages in the test and in the assistant |
| Empty or meaningless chunks, "Index quality" with a warning | A scan without OCR or weak recognition | Choose a PDF parser with OCR, check the chunks, process again |
| Similar chunks from neighboring documents outrank the right one | There is no distinguishing feature | Connect a tag set or set metadata, see Metadata, tags and PageRank |
| The document exists, but it is never found | The document is disabled or does not have the "SUCCESS" status | Check the document list |
Checking through the API
The same search is available through the POST /api/v1/retrieval method with an organization key. The main request fields are question, dataset_ids, the optional document_ids, similarity_threshold, vector_similarity_weight (the share of the vector part; the interface sets the inverse value), top_k, rerank_id, keyword, highlight, use_kg, cross_languages, page and page_size. All bases in the request must use one embedding model. The response contains chunks with the fields similarity, vector_similarity, term_similarity, document_id and dataset_id, which is convenient for automatic regression: a set of control questions with the expected documents is run after every reprocessing of the base. How to get a key: API key and SDK.
Before connecting to an assistant
The check is considered passed when the right source is among the first results for all control questions, the numbers and names are not distorted, and the negative question does not produce a false confirmation. After that, create an assistant, transfer the selected parameters into it and repeat the same questions in chat. The assistant adds a model and a prompt to the search, so its answer should be checked separately, see Chat with sources.
Ask Sema