Local AI and RAG : A Guide to Using Models on Your Data

Local AI and RAG : A Guide to Using Models on Your Data
13 min read

Running a language model on your machine and having it answer from your documents are two distinct projects that combine. Local AI keeps queries on your hardware; RAG, or retrieval-augmented generation, fetches useful passages from your files before drafting the response. With 16 GB of memory, a launcher like Ollama or LM Studio, and a quantized model of a few billion parameters, you can already test both without a subscription.

This guide deliberately separates the two paths. The first answers the question ‘how do I run a model at home?’: hardware, model choice, tools. The second answers ‘how do I query it on my data without it inventing?’: document splitting, search, citations, and control. The two converge in a common architecture, described below. The versions and figures cited were verified in November 2026 from official documentation; they evolve quickly and should be rechecked before any hardware purchase. To go further, read our article ‘SQL FULLTEXT Search vs Vector Database: Which Engine for RAG?’.

Why run AI locally?

The first reason is data control. When the model runs on your workstation or an internal server, the text you submit to it does not need to transit through a third-party service. Ollama’s documentation specifies that in local execution, the tool does not have access to your prompts, and cloud functions can be disabled by the OLLAMA_NO_CLOUD=1 variable or a setting in ~/.ollama/server.json (Ollama FAQ). LM Studio states that conversations and documents attached for RAG remain on the machine, with Internet only required to search and download models or updates (LM Studio offline documentation).

Next are cost predictability and stability. A local model does not charge per token: you pay for hardware and electricity, regardless of volume. You also freeze the model version, preventing a provider-side update from changing the behavior of an already validated application. Finally, execution works offline once files are downloaded, an argument for travel or isolated sites.

However, compromises must be accepted. Models that fit on a laptop are generally less performant than large commercial API models on complex reasoning, and speed depends directly on your graphics card or chip. Installation, updates, and service security also become your responsibility.

Essential vocabulary

  • Size: the parameters learned by the model, distributed as multi-gigabyte files. Their number (8B for 8 billion, for example) largely determines the required memory.
  • Quantization: the reduction of the precision of these weights (from 16 bits to 8 or 4 bits, for example), which divides the file size at the cost of a more or less noticeable quality loss.
  • Context window: the maximum number of tokens (word fragments) the model can process at once, including question, history, and documents.
  • Open weights and open source: a model whose only weights are published is ‘open weights’. The Open Source Initiative’s open source AI definition (version 1.0) requires more: detailed information on training data, full training and inference code, and parameters, all under OSI-approved conditions (Open Source AI Definition). Most models you install are therefore open weights models, under more or less permissive licenses.

Hardware, models, and tools to get started

Memory is the first criterion, before computing power. A model must fit entirely in the video memory (VRAM) of the graphics card, or in the unified memory of an Apple silicon Mac, to run at a comfortable speed. What overflows is offloaded to RAM and the CPU, which works but slows significantly. The total need breaks down into three items: the weights, the key-value cache (KV cache) that grows with context length, and a margin for software and the system.

Google publishes a useful table for its Gemma 4 family to set orders of magnitude. It indicates the memory needed just to load the weights, with about a 20% margin included but without the KV cache (Google Gemma documentation, consulted November 2026).

Gemma 4 Model 16 bits (BF16) 8 bits (SFP8) 4 bits (Q4_0)
E2B 11.4 GB 5.7 GB 2.9 GB
E4B 17.9 GB 8.9 GB 4.5 GB
12B 26.7 GB 13.4 GB 6.7 GB
26B A4B (expert) 57.7 GB 28.8 GB 14.4 GB
31B 69.9 GB 34.9 GB 17.5 GB

Two lessons emerge from this table. First, 4-bit quantization divides the footprint by about four compared to 16 bits, which explains why launchers almost always offer quantized versions by default. Second, an expert model (MoE) like the 26B A4B only activates 4 billion parameters per token, but Google specifies that all 26 billion must still be loaded into memory: the architecture speeds up computation, not storage.

Three typical configurations

These figures allow reasoning about concrete cases, to be refined with the site’s local AI model calculator, which estimates memory usage and speed for a given card or Mac:

  • 16 GB laptop without a dedicated graphics card: aim for models of 2 to 4 billion effective parameters in 4 bits (less than 5 GB of weights for Gemma 4 E4B), with a modest context. This is sufficient for rephrasing, summarizing, or extracting information from a short text.
  • PC with an 8 GB VRAM card: a model in the 8 to 12 billion parameter class in 4 bits (6.7 GB of weights for Gemma 4 12B) fits, but the remaining margin for the KV cache is small. Reduce the context or accept partial overflow to RAM.
  • Mac or workstation with 32 GB of available memory: models of 26 to 31 billion parameters in 4 bits become accessible (14.4 to 17.5 GB of weights for Gemma 4), with room for a longer context.

On the tool side, two launchers dominate for initial use. Ollama, published under the MIT license, works via command line and application, and exposes a local API on port 11434; its latest stable version on GitHub was v0.40.16, released November 9, 2026. LM Studio, free including for professional use since July 2025, prioritizes a graphical interface, supports GGUF and MLX formats, and also offers a server compatible with the OpenAI API. Our Ollama vs LM Studio comparison details their differences, and our step-by-step Ollama tutorial covers installation and the API.

For models, remember four criteria: size compatible with your memory, quality in French, target task (writing, code, extraction, vision), and license. Several recent families are published under the Apache 2.0 license, which permits commercial use: Gemma 4 from Google, Qwen3.5 from Alibaba, Ministral 3 from Mistral AI, and gpt-oss from OpenAI. Our selection of the best local AI models compares them by use and machine.

What is RAG and when is it needed?

RAG (retrieval-augmented generation) combines a language model with a document search engine. The term comes from an article by Patrick Lewis and his co-authors, submitted in May 2020, which described models combining a ‘parametric’ memory, that of the weights, and a ‘non-parametric’ memory, a document index queried on demand (Lewis et al., 2020). In current practice, the principle remains the same: before answering, the system searches for relevant passages in your corpus, inserts them into the prompt, then asks the model to rely on them. To delve deeper, consult ‘Chunking RAG: How to Split Your Documents for Better Answers?’.

This approach addresses two limitations of models alone. An LLM does not know your internal documents, and its knowledge stops at its training date. RAG provides it with up-to-date, verifiable content at the time of the question, which also allows demanding citations. It does not modify the model: adding or removing a document is enough to update what it ‘knows’.

A decision grid

RAG is not always necessary. Depending on corpus size and update frequency, a simpler solution may suffice. The following grid summarizes the most common situations; it reflects our editorial analysis, not a standard.

Situation Simplest Solution Why
One or two short documents to analyze occasionally Paste text into the prompt The content fits within the context window, no indexing is useful.
Hundreds of documents, varied questions RAG Only relevant passages are sent, saving context and allowing source citation.
Documents updated often (procedures, prices, FAQ) RAG Re-indexing is sufficient; the model does not change.
Need for a consistent tone, format, or business vocabulary System prompt, then possibly fine-tuning This is a behavior to learn, not knowledge to retrieve.
The system must act (send an email, modify a file) Agent with tools Document search is no longer enough; permissions and supervision are needed.

The boundary between RAG and model tuning is the subject of a dedicated article in the series, RAG vs fine-tuning. As for the last case in the table, it belongs to another logic: if your project involves giving the model tools to act, our AI agents guide explains how to design and oversee them. We detail this point in ‘RAG vs Fine-tuning: How to Choose Between the Two?’.

How to build reliable document search?

RAG is built in two stages. Indexing prepares documents once and for all; querying occurs with each question. Final quality depends much more on the first step than on the choice of generation model: a poorly split or unfindable passage will never be cited correctly, regardless of the LLM.

  1. Extract text from PDFs, web pages, or office files, preserving titles, source, and date of each document. Tables and scanned PDFs often require specific processing.
  2. Split text into passages (chunking), by fixed size, paragraph, or section. The right granularity depends on document type; our article on choosing a chunking strategy offers a decision tree.
  3. Index passages: either by keywords (classic full-text search), by vectors (embeddings), or both. Ollama exposes, for example, an /api/embed endpoint that returns normalized vectors; its documentation recommends cosine similarity and using the same embedding model for indexing and queries (Ollama embeddings documentation).
  4. Search for passages closest to the question, then optionally re-rank them.
  5. Generate the answer by asking the model to rely solely on the provided passages, cite their source, and answer ‘I don’t know’ if information is missing.

The choice between full-text search and vector database is not ideological. Keyword search excels on exact references (an article number, a product code), vector search on rephrasing and synonyms. Many systems combine both. The series dedicates an article to comparing SQL FULLTEXT search and vector databases, and another to embeddings and semantic search. The subject is covered in a dedicated article: ‘Embeddings: Understanding Vectors and Semantic Search’.

Where RAG most often fails

  • Context too short: below 24 GiB of VRAM, Ollama allocates a default window of 4,096 tokens according to its documentation. Ten passages of 400 tokens, plus the question and instructions, already saturate it. Increase it (OLLAMA_CONTEXT_LENGTH variable or num_ctx parameter) considering the additional memory.
  • Passages split at the wrong place: a definition separated from its title, a table split across two pieces.
  • Embedding model poorly suited to French or different between indexing and query.
  • Absence of citations: without displayed source, the user cannot distinguish a founded answer from an invention.
  • No set of reference questions: without regular evaluation, a regression goes unnoticed after a corpus or model update.

For this last point, compile about thirty questions for which you know the answer and source document, then verify with each change that the correct passage is found and that the answer cites it. Our guide to evaluating a language model details applicable measurement methods. The complete implementation, from corpus to interface, will be covered in a tutorial to create a RAG chatbot on your documents.

Security, costs, licenses, and limits

Local does not mean secure by default. Ollama listens by default on 127.0.0.1, i.e., only on the machine itself; making it accessible to the network via OLLAMA_HOST=0.0.0.0 exposes an API without native authentication to all workstations that can reach it. If several people must use it, place it behind an authenticated proxy or firewall. Also note that automatic updates and model downloads generate outgoing traffic, to be documented if your security policy requires it.

RAG adds its own risks. OWASP ranks prompt injection at the top of its LLM Application Top 10 2025 and emphasizes that malicious instructions can arrive from external sources, like files or web pages, and that RAG does not remove this risk (OWASP, LLM01:2025). A booby-trapped document in your corpus can thus influence responses. The other risk is leakage via context: if the index contains HR documents, a user without clearance must not be able to retrieve them via a clever question. Access rights filtering must apply at search time, not just in the interface.

ANSSI published security recommendations for generative AI systems in April 2024 that apply directly to these projects (ANSSI-PA-102 guide). These notably include using secure model formats (R6), considering need-to-know from design (R8), filtering inputs and outputs (R25), limiting automatic actions triggered from uncontrolled inputs (R27), and logging processing (R29). An article in the series will be dedicated to RAG system security.

On the cost side, the equation is simple to state but not to decide. Local replaces a usage-based bill with hardware investment and administration time; it becomes interesting when volumes are high and regular, or when confidentiality prohibits the cloud. For occasional use, a token-charged API often remains more economical and performant. The site’s interactive model comparator helps situate commercial models against this alternative.

Licenses, finally, vary greatly from model to model. Gemma 4, Qwen3.5, Ministral 3, and gpt-oss are published under Apache 2.0. Mistral Large 4, whose weights were published in late October 2026, falls under a custom license derived from the MIT license, requiring a separate commercial license to exploit it as a hosted service (model-as-a-service), according to its Hugging Face card. Always read the license before commercial use or redistribution.

Check your installation in five points

  1. The service only listens on the intended address (local or protected internal network).
  2. The ollama ps command, or your launcher’s equivalent, shows the model loaded 100% on the GPU if that’s the goal, and the context length actually allocated.
  3. No cloud function is enabled if your data must not leave the machine.
  4. The model’s license is compatible with your use, and its version is noted.
  5. A set of reference questions allows measuring quality before and after each modification.

This check does not guarantee the absence of flaws: it establishes a controllable baseline, to be revisited with each tool or model update.

Tutorials and sizing tools

This guide serves as an entry point to a series of practical articles, published gradually. The ‘local execution’ path starts with installing a local AI on Windows, macOS, or Linux, continues with an Ollama tutorial dedicated to commands and the API, then with the Ollama vs LM Studio comparison. It is complemented by the selection of the best local AI model by task and a guide on the RAM and VRAM needed for a local LLM, with quantified simulations. We detail this point in ‘Which AI Model to Choose? Guide and Comparison of LLMs’.

The ‘questions on documents’ path starts with a detailed definition of what RAG is and its architecture, then addresses the choice between RAG and fine-tuning, embeddings, chunking, the comparison between FULLTEXT and vector databases, creating a RAG chatbot on your documents, and RAG security. The two paths can be followed independently. To go further, read our article ‘RAG: Definition, Architecture, and Concrete Examples’.

Measure before buying

Before investing in a graphics card, test with what you have. Install a launcher, load a small quantized model, then note three values: memory used, generation speed in tokens per second (Ollama provides it via the eval_count and eval_duration fields of its API), and answer quality on about ten questions representative of your use. Then compare with the site’s calculator estimate for the considered machine.

This method has its limits: speeds vary by driver, launcher version, and model format, and quality in French is not found in any memory table. However, it has the advantage of relying on your own data, which remains the best indicator for a local AI or RAG project.

S’abonner
Notification pour
guest

0 Commentaires
Le plus ancien
Le plus récent Le plus populaire
0
Nous aimerions avoir votre avis, veuillez laisser un commentaire.x