Not every search needs an LLM

The question behind the question

Hardly any conversation about your own knowledge gets by today without the question of how to connect an LLM to it. The documentation should become searchable, support should find answers faster, and customers should be able to ask a question inside the product instead of clicking through menus. That is a fair wish, and large language models are impressive tools.

As soon as it gets concrete, though, the same questions come up. What does each single request cost when it runs through an external API? How long does someone wait for the answer? Are the questions of our users and the contents of our knowledge even allowed to go to a third-party provider? And what happens when the model invents an answer that sounds plausible but appears in none of our texts?

These questions often disappear once you look more closely at the actual goal. In many cases nobody wants text to be generated; they want to find the right place in the knowledge that already exists. That is a search task, and for search tasks there are tools that are much smaller, faster and cheaper than a large language model.

LLM, SLM, RAG and embeddings in brief

An LLM, a large language model, is a language model with billions of parameters that understands text and produces new text. It usually runs at a provider on GPUs, because it is too large to operate on an ordinary server. An SLM, a small language model, works on the same principle but is small enough to run on your own hardware or even a laptop. In return it is less versatile and makes mistakes sooner on hard tasks.

RAG stands for retrieval-augmented generation and describes a pattern rather than a model. Before the language model answers, a system picks the matching passages out of your own knowledge and hands them to the model as a basis, so that it answers from your texts instead of from memory. The decisive part of RAG is therefore the search that comes first, because the model can only answer as well as the passages it was given fit the question.

Embeddings are exactly the tool for that search. An embedding model does not produce text; it translates a text into a list of numbers that describes its meaning. Such models are orders of magnitude smaller than an LLM. This leads to a sober insight: if you mainly want to find things, you often only need the first step of RAG and no language model at all to phrase the result.

Level one: keywords and an index

Everyone knows the lightest form of search. You split headings, summaries and body text into single words, drop filler words such as "and" or "the", and record in an index which word occurs how often in which document. A proven method like BM25 weights rare words more heavily than common ones, because "DNS" tells you more about a text than "server".

Such an index for a few hundred pages takes up only a tiny share of memory and answers a request practically instantly. It needs no model, no GPU and no external service. For many products this already makes a good search, and it stays in place as the foundation for the levels that follow.

The drawback lies in the nature of the thing, because a keyword search compares words, not meanings. Someone who types "the cloud bill is exploding" only finds the article on cost control if exactly those words appear in it. A word index captures the intent behind a question only as far as the person searching happens to use the same terms as the person who wrote the text.

Level two: embeddings understand intent

An embedding model reads a text and returns a vector, a fixed number of numbers. With the model we use, there are 384 of them. You can picture every text as a point in a space with 384 directions, and the model is trained so that texts with a similar meaning lie close together, even when they share not a single word. "The cloud bill is exploding" and "How do we lower our AWS costs?" therefore end up in the same neighbourhood.

To search, you translate the question into a vector as well and compare it with all stored vectors. The measure for this is cosine similarity, which tells you how far two vectors point in the same direction. When all vectors are normalised to length one, this boils down to a simple dot product, and comparing against the roughly 500 chunks per language of this compendium takes well under a millisecond on an ordinary CPU. At this size there is no need for a dedicated vector database.

What matters is that the expensive work happens beforehand. The content is vectorised once, and the result ships as a file. At runtime the model only has to translate the one short question. Our model is a quantised multilingual-e5-small in ONNX format which, after trimming it to Latin script, is about 57 MB in size and takes up roughly 275 MB of memory once loaded. It runs on a single CPU core without a GPU, and after the first load a question is answered in a few milliseconds.

The workflow: building embeddings locally

The entire process runs on your own machine, so neither the content nor later search queries ever reach an external service. It consists of a few steps that a single script runs one after another.

First, the model is downloaded from Hugging Face exactly once, at a pinned revision whose files are checked by SHA-256. Because we only serve German and English, we trim the vocabulary to Latin script, which halves the memory footprint without changing how German or English text is tokenised. The trimmed model is checked into the repository, so neither the build nor the running server ever downloads anything.

Next, the content is cut into chunks. Each article contributes its summary, its sections, which we pack into chunks of at most about 1,200 characters, and every FAQ answer as a chunk of its own. A chunk should be about exactly one thing, because that is what makes its vector precise. Heading and title go in front, so that even a paragraph that only says "this" or "it" still carries what it is about. Figures contribute their caption, and code is skipped because listings vectorise poorly.

Each chunk is then vectorised with the prefix "passage:", and a question later with "query:", because the model was trained on exactly these pairs. For every chunk we store a hash of model and text. If only one paragraph changes, only that one is recomputed and everything else is taken over from the last run, so a run usually takes only seconds. The result is a JSON file of about 3.4 MB that is checked in together with the code.

npm run embeddings

# 1. Check the model: pinned revision, SHA-256 per file, trimmed
# 2. Cut articles into chunks: summary, sections, FAQs, intents
# 3. Vectorise only changed chunks (hash of model and text)
# 4. Write embeddings.json: one chunk per line, 384 numbers per vector

At runtime the file is loaded and validated once, and the model is only brought into memory with the first question. So that the server does not get overloaded by many simultaneous questions, it computes only one question at a time and allows no more than a short queue. When the queue is full or the model is unavailable, the keyword search answers on its own, so the page is never left empty.

Combining keywords, intents and aliases

Embeddings are strong on intent but weaker on exact terms. Someone searching for "BM25" or a specific product name wants to find exactly that word, not a text that means roughly the same. That is why the second level does not replace the first but adds to it. Every question runs through the keyword search and the vector search in parallel, and each returns its own ranking.

The rankings cannot be added up directly, because a BM25 score and a cosine similarity live on entirely different scales. We therefore use reciprocal rank fusion, where only the rank counts. A page receives a contribution from each list that is larger the higher it ranks there, and whatever appears high up in several lists ends up at the very top. Each list first filters with its own threshold, so that a question about the weather does not find some article after all.

The intents are added as a third list. For each article we write a few sentences with which someone describes their problem before they know that this article is the answer, such as "something breaks after every deployment". These sentences are vectorised like questions and never shown. A question that resembles one of them pulls the matching article forward, even if the article itself uses quite different words.

That leaves the problem of jargon. Your own team may say "KI", the English documentation says "AI", and some people spell out "artificial intelligence". For the embedding this is no problem, because it recognises the terms as equivalent anyway, even across languages. The keyword search, on the other hand, sees three different words. It therefore folds such aliases into one shared term before indexing, so that "AI" also finds texts that say "KI", and it counts the term only once, so that a synonym does not artificially inflate the score.

The same principle works for any internal jargon. If your team says "tenant" while your customers write "account", or if a product carries an internal code name, exactly that mapping belongs in a small, maintained alias list. In addition, our articles carry their own keywords for terms that appear neither in the title nor in the summary, such as a tool or a standard an article is built on. They are searched but never shown.

When an LLM is worth it after all

These levels do not replace a language model where something genuinely has to be written. When an answer has to be assembled from several sources, summarised or carried over into a new context, an LLM is the right tool, and RAG with a good search in front of it is then the clean way to go. Even there, though, the quality of the search decides whether the answer is right.

For many use cases, however, it is enough to lead people directly to the right place. That is faster, because no answer has to be generated, cheaper, because no request goes through a paid API, more confidential, because no question leaves your own server, and more honest, because nothing can be made up. What is shown really is in the text.

This is exactly how the search in this compendium works. Ask it a question at Ask the compendium in your own words, with a typo, in German or with "KI" instead of "AI", and see where you land. If you want to build a search like this for your own product or knowledge, get in touch.