A tutorial details the construction, in Python, of a multimodal retrieval-augmented generation (RAG) pipeline capable of searching and reasoning on both text and images extracted from complex documents. The principle targets PDFs that mix prose, tables, and charts, formats where a classic RAG struggles based on text alone.
RAG involves retrieving excerpts from a document to pass them to a language model before it formulates its answer. Applied to text alone, the process yields good results. However, segmentation based solely on text destroys the formatting of tables and leaves out images, so the system responds from an incomplete view of the document. The described pipeline, in contrast, treats images as queryable objects on the same footing as text.
The implementation relies on LangChain, the unstructured library, and OpenAI’s API. The tutorial describes four steps: analyzing the document, summarizing images, storing everything, and querying the system. The poppler-utils system package, a dependency, is required for unstructured to extract images from PDFs.
Separating text and images
The partition_pdf() function from unstructured reads the document and classifies each element by type: paragraphs, tables, and embedded images. The images are saved in a dedicated folder, while text blocks are grouped into fragments of a maximum of 4,000 characters, based on headings. A loop then separates the text fragments from the image file paths.
A classic vector database cannot index an image directly. Therefore, each image is passed to GPT-4o with the instruction ‘Describe this image in detail for use in a search system,’ to obtain a detailed description that serves as a queryable substitute. The tutorial notes a cost at this stage: each image triggers an API call. For documents containing many logos or icons, it suggests filtering out the smallest images before this processing.
A two-tier storage system
The architecture relies on a MultiVectorRetriever, which manages two storage spaces. The first, a Chroma vector database, contains the text fragments and image descriptions, transformed into vectors using the text-embedding-3-small embeddings model. The second, an in-memory document store, retains the raw content. Each entry is linked to its counterpart by a unique identifier. Once the search is performed on the descriptions, the system thus retrieves the original text fragment or image file.
During a query, the question itself is vectorized and compared to the indexed elements. The text fragments are passed as-is, while the retrieved images are opened and then encoded in base64, a text format that allows them to be attached to the query. The whole is sent to GPT-4o, which produces an answer based on both types of content. The tutorial illustrates the approach with the question ‘What does the revenue chart show for Q3?’, which can therefore be used as-is as an example.
Paths for evolution
The author mentions several possible extensions: adding a re-ranking step to improve search accuracy, saving the Chroma database to disk for production use, or exploring vision-language models like ColPali, which directly integrate document pages as images and thus bypass the analysis step.
