Uploading documents
Which files a knowledge base accepts, what processing does with PDF, Word, Excel, presentations and images, how to read statuses and progress messages, when to reprocess a document and where to view chunks.
A document becomes available to an assistant only after it is uploaded and successfully processed. This article describes the path of a file from the "Add file" button to the list of chunks.
Uploading
- Open "Dataset", choose a base and go to the "Docs" tab.
- Click "Add file", select files in the file explorer or drag them to the upload area. You can add several files, and each one is uploaded separately; the interface limits a single file to one gigabyte.
- Check the names and sizes and click "Upload".
- If the upload window has a "Parse immediately" switch, leave it on so that processing starts automatically. Otherwise, select the documents in the list and click "Parse".
There is a second way: upload a file to the "File Management" section and link it to one or several bases with the chain icon "Link to Knowledge Base". Then, when you delete a base or a document, the source file stays in the file storage.
Tip
Before uploading, check that the right organization and the right base are selected, that the document is not password-protected and that the same file has not been uploaded yet. Do not upload a corrected version under the old name without a version mark if the old document must stay in the base.
Supported formats
| Format | What processing does |
|---|---|
| The PDF parser recognizes the page structure, tables and the text of scans (OCR). For purely text PDFs you can choose the "Naive" mode: it is faster, but it takes only the text layer | |
| Word (DOC, DOCX) | Text, headings and tables are extracted. Word is processed separately from PDF, and the PDF parser settings do not affect it |
| Excel (XLS, XLSX, CSV) | By default each row is turned into "column header: value" pairs. For complex tables with merged cells, turn on "Excel to HTML" |
| PowerPoint (PPT, PPTX) | Slide text; with the "Presentation" method each slide becomes a separate chunk |
| Text, Markdown, HTML, JSON | Text is split by delimiters and by the chunk size threshold |
| Emails (EML) | Parsed as an email: headers, body, attachments |
| Images (JPEG, PNG, TIF, GIF and others) | Text is captured by OCR; the image description is built by an image recognition model if one is configured in the organization |
| Audio | Text is produced by a speech recognition model if one is configured |
If the format is not recognized, the upload ends with a format error message. What exactly happens to a file is determined by the chunking method of the base or the document, see Chunking methods.
Processing statuses
The document list has the columns "Name", "Chunks", "Size", "Upload Date", "Chunking method" and "Parsing Status".
| Status | Meaning | What to do |
|---|---|---|
| PENDING | The document is uploaded, and processing has not started or is in the queue | Click "Parse" or wait |
| PARSING | Parsing, cutting and indexing are in progress | Wait for it to finish; a document cannot be deleted at this time |
| SUCCESS | The chunks are written to the index | Check the number of chunks and run a control search |
| CANCELED | Processing was stopped by the user | Start it again if needed |
| FAIL | Processing was interrupted | Open the progress messages and pass them to the base owner |
Only documents with the "SUCCESS" status and the switch turned on in the state column ("Enable") take part in search. A disabled document stays in the base but does not appear in answers.
Progress messages
Each document has the fields "Begin at", "Duration" and "Progress". The progress shows step by step what processing is doing: page recognition, creating chunks with their count, building vectors, writing to the index and the total time. When additional features are turned on (keywords, questions, tags, knowledge graph, RAPTOR), separate steps appear and processing takes noticeably longer.
After indexing, a quality estimate is shown: "Index quality: N/M suspect chunks" with the reasons (empty, empty tables, garbled, too short, mostly symbols). A high share of suspect chunks comes with a warning that advises repeating the processing, for example by turning on full-page OCR or changing the table processing mode. A message with an error prefix gives the reason for the stop: the allowed size is exceeded, the file is not found in the storage, the task is canceled.
Reprocessing
Changing the base settings does not change chunks that already exist. After changing the PDF parser, the chunk size or the method, select the documents and click "Parse". The platform asks you to confirm clearing the existing chunks; after that the old chunks are deleted and new ones are created.
You can change the method for a single document without touching the base: in the document actions open "Chunking method", choose a method and parameters (for PDF, for example, a page range) and save. Before reprocessing, write down the current number of chunks and the result of a control search so that you can compare quality.
Deletion and replacement
A document is deleted from its actions with the "Delete" button and a confirmation. The chunks of the document are removed from the index, and assistants stop referring to it. If the document took part in the knowledge graph, the related nodes are deleted, but the graph itself is updated only when the next document is processed.
To replace a document with a new version, upload the new file, wait for the "SUCCESS" status, check it with a control search and only then delete the old one. That way the base never has a period without the document.
Warning
Deletion and reprocessing change the answers of all assistants that use the base. Warn the assistant owners before bulk actions.
Viewing and editing chunks
Click the name of a processed document: the chunks page opens. Here you can:
- switch between the "Full text" and "Ellipse" modes and search across chunks;
- turn individual chunks on and off, including in bulk ("Select All", "Enable selected", "Disable selected", "Delete selected");
- open a chunk with a double click, edit the text, add a "Keyword" or a "Question". Keywords raise a chunk in the results for queries with these words; the questions you set are used when the vector of the chunk is built.
Check the beginning, the middle and the end of the document, the tables (headers, numbers, units of measurement) and the quality of the recognized text for scans. The whole document opens with the view button and lets you match a chunk with the source page.
How to check that the base finds the right chunks: Checking search on a knowledge base.
Ask Sema