AI Knowledge Assistant: the complete guide

Every company past a certain size is sitting on the same invisible problem: the answer to almost any operational question exists somewhere inside the organisation, but finding it reliably is slow, inconsistent, and depends too much on knowing the right person to ask. A knowledge assistant is how you make that knowledge reliably accessible without rebuilding your information architecture.

What a knowledge assistant is

A knowledge assistant is an AI system that answers questions about your organisation's specific information. It is not trained on that information the way a language model is trained on the internet. Instead, it reads from your sources at query time, retrieves the passages most relevant to a question, and uses a language model to compose an answer from exactly those passages, with a citation to the source so the answer can be verified.

It is worth being precise about what it is not. It is not a search engine, because it gives you a composed answer rather than a list of links. It is not a generic AI chatbot, because it only answers from your documents and will tell you when it cannot find a relevant source rather than making something up. And it is not a training exercise: you do not fine-tune a model on your knowledge base. You retrieve from it.

The distinction matters because retrieval and training have opposite tradeoffs. Training is expensive, slow to update, and can bake in outdated information. Retrieval is cheap, instantly updatable, and transparent: every answer has a traceable source.

What RAG means in plain terms

RAG stands for Retrieval-Augmented Generation. The name is more intimidating than the concept. Here is the idea in plain terms:

When a user asks a question, the system does not immediately pass it to a language model. First it searches the index for the passages most likely to contain the answer. Think of it as finding the right pages in a very large book before asking someone to summarise them. Those passages, called the context or the retrieved chunks, are then handed to the language model along with the original question. The model composes its answer from what is in the context, not from its training data.

The result is an answer that is specific to your documents, citable because the source passages are known, and bounded by what your knowledge base actually contains. If the answer is not in your documents, a well-built assistant says so rather than guessing.

The free pilot

We open every engagement with a free pilot. You connect one or two of your existing sources: a Confluence space, a Google Drive folder, a Notion database. We index them, build the retrieval pipeline, and then you test the assistant on the questions your team actually asks every day.

At the end of the pilot you have a measured accuracy figure on your real knowledge, not a vendor's claim about what the technology can do in theory. You also have a citation rate, the share of answers that link back to a specific source, and a gap analysis showing which question categories are underserved. You keep all of that whether or not you proceed with the full build.

The pilot is free because it is the only honest way to scope the work. The right number of sources to index, the right chunking strategy, the right retrieval depth, and the achievable accuracy all depend on the specific shape of your knowledge. A measured pilot tells you exactly what you are buying.

How retrieval works

The retrieval step is where most of the interesting engineering lives. When a question comes in, the system needs to find the relevant passages from an index that may contain millions of pages. It does this in three sub-steps: chunking, embedding, and vector search.

Chunking is dividing your documents into passages of a manageable size, typically a few hundred words each, while trying to preserve semantic coherence. A chunk that cuts across a paragraph boundary mid-sentence is harder to retrieve accurately than one that respects the natural structure of the document. Good chunking is a craft, not a setting to dial in.

Embedding is converting each chunk, and each incoming query, into a vector: a list of numbers that captures the meaning of the text in a way that allows mathematical comparison. Two pieces of text that are semantically similar will have vectors that are close together in this high-dimensional space, even if they share no words in common. That is what allows the assistant to find a policy document in response to a question phrased very differently from the document's own language.

Vector search finds the chunks whose embedding vectors are closest to the query vector. This is done with a vector database, a system optimised for the kind of approximate nearest-neighbour search that retrieval requires at scale. The top results, typically the ten to twenty most relevant chunks, become the context for the language model.

The RAG pipeline: from question to cited answer

Grounding and citations

Grounding is what separates a knowledge assistant from a generic chatbot, and it is the most important concept in this guide. Every answer the assistant produces is composed from specific retrieved passages. Each of those passages is traceable to a specific document, page, and section. The assistant includes that citation in its answer, and a user can click through to the exact source.

Why does this matter so much? Because a language model that is not constrained to specific sources will fill in the gaps from its training data, and in a business context those gaps are exactly where the most sensitive and variable information lives: your current policy, your latest pricing, your specific process for a given market. Grounding prevents that. The assistant answers from what your documents actually say, not from what the model was trained to say about how companies generally operate.

Citations also change how the answer lands with the person asking. An answer without a citation asks for trust. An answer with a citation to the specific policy page and paragraph invites verification. That difference matters enormously for the use cases where a knowledge assistant has the highest value: onboarding, compliance, support and sales.

There is an operational benefit too. When an answer is wrong, a cited answer tells you exactly where the problem is: either the citation is wrong, which is a retrieval problem, or the citation is right but the document itself is wrong, which is a content problem. An uncited answer just leaves you with a wrong answer and no path to fixing it.

Permission-aware answers

One of the first questions every team asks is: what happens when someone asks about something they are not supposed to see? A well-built knowledge assistant enforces access control at the retrieval layer, not as a post-processing step.

When the assistant retrieves passages for a question, it filters the index by what the person asking is allowed to read. A new hire does not have the finance team's restricted files surfaced in their answers, because those files are excluded from their retrieval pool before any search happens. The same document can appear in one person's answers and not in another's, based on the same permission model that governs the original source.

Permission-aware retrieval: each user only sees allowed sources

In practice, this means the permission model of your knowledge assistant mirrors the permission model of your existing sources. Confluence space permissions, Google Drive sharing settings, Notion page access: these translate directly into retrieval filters. You do not need to build a new access control system. You inherit the one you already maintain.

This is also why a knowledge assistant is safer than most alternative approaches to surfacing internal knowledge. A shared wiki page that someone searches directly might reveal its existence through search results even if the page itself is restricted. In a permission-aware retrieval system, the page is simply not in the retrieval pool for that user, so there is nothing to surface.

Where the models run

There are two classes of model in a knowledge assistant: the embedding model that converts text to vectors, and the language model that composes answers from retrieved chunks. Both can run in the cloud or inside your own infrastructure.

For most teams, running on a hosted cloud provider is the right starting point. It is faster to set up, easier to scale, and the models available through cloud APIs are excellent. The tradeoff is that your document content travels to the provider's infrastructure when it is embedded and when it is included in a prompt.

For teams with strict data residency requirements, sensitive regulated data, or security policies that prohibit external data transfer, we run both models inside your own environment. This typically means running an open-weights embedding model and a smaller open-weights language model on your own servers or your private cloud tenancy. The tradeoff is more infrastructure to manage and usually a somewhat slower or less capable model, though the gap has narrowed considerably.

Keeping the index fresh

A knowledge assistant is only as useful as its index is current. If an HR policy was updated six months ago but the index still reflects the old version, the assistant will confidently give outdated answers with citations that look right but are not.

Index freshness is maintained through sync. Most source systems expose webhooks or change notification APIs that allow the index to be updated as soon as a document changes. Confluence notifies on page edits. Google Drive notifies on file changes. Notion exposes a changelog. Where webhooks are not available, a scheduled crawl re-indexes changed content on a configurable interval, typically daily.

The re-indexing process for a changed document is efficient: the system identifies which chunks are affected, removes the outdated vectors from the index, and adds the new ones. A large knowledge base can typically be kept current with minimal latency, so an updated policy page is answerable from its new content within minutes of the change.

This is the mechanism that makes a knowledge assistant more useful than a static documentation portal: the portal requires someone to remember to update it and then check it; the assistant answers from whatever your source systems currently say.

Evaluating accuracy

Accuracy in a knowledge assistant means two different things, and it is worth tracking both separately.

Retrieval accuracy is whether the system found the right passages. If the retrieved chunks do not contain the answer to the question, the model cannot give a correct answer even if it is excellent at composition. You measure this by asking questions with known answers and checking whether the relevant source passages appear in the top retrieved results.

Answer accuracy is whether the model composed the correct answer from the passages it retrieved. A model can retrieve the right passage and still misread it, or retrieve a partially relevant passage and produce a technically true but misleading answer. You measure this by comparing the model's answer against a ground truth answer from a test set of real questions from your team.

Citation rate is a third signal worth tracking: what fraction of answers include a citation to a specific source. A high citation rate is not sufficient for accuracy, but a low citation rate is a reliable signal that something is wrong with either the retrieval or the grounding.

The no-answer rate deserves a note. A well-calibrated assistant should decline to answer when it cannot find relevant sources. If the no-answer rate is zero, the assistant is probably guessing on questions outside its knowledge rather than saying it does not know. That is a sign of over-confidence that will undermine trust.

Channels

Where the assistant surfaces matters as much as what it can do. The best channel is the one your team already lives in.

Slack is the most common first deployment. Users ask questions in a dedicated channel or by mentioning the assistant in any thread. The assistant replies in the thread with its answer and citations. Slack's threading model makes it easy to follow up, share the answer with a colleague, and refer back to it later.

Microsoft Teams works the same way: a Teams bot that answers questions in channels or in direct message, with citations that link back to the source document in SharePoint or wherever it lives. For organisations that run on Microsoft 365, Teams integration also means the assistant can be made available inside specific team channels, limiting its scope to the knowledge most relevant to that team.

A web widget is the right choice when you want the assistant available on an internal web page or portal, like a support or onboarding site, without requiring everyone to use Slack or Teams. The widget is embedded with a few lines of JavaScript and appears as a floating chat element.

The underlying assistant is the same regardless of channel. The channel integration is a thin layer on top of the retrieval and composition pipeline.

Integration

A knowledge assistant is only as good as the sources it can reach. The integration layer is how those sources connect to the index.

For most common tools, connectors already exist: Confluence uses the REST API to pull pages and attachments, Google Drive uses the Drive API to read documents and Sheets, Notion uses its public API, SharePoint uses the Microsoft Graph API. For Jira and Zendesk, the APIs expose tickets and knowledge articles. GitHub exposes repositories and wikis. For internal databases or bespoke systems, we write a custom connector against whatever interface is available.

The connectors handle three jobs: initial bulk indexing, incremental updates as content changes, and permission metadata so the retrieval layer knows who is allowed to see what. The last one is the most important to get right and the most commonly neglected in off-the-shelf solutions.

PDFs are a special case. PDFs that are text-based index cleanly. PDFs that are scanned images need OCR first. For teams with a mix of both, we run an OCR pass as part of the indexing pipeline so scanned documents are queryable alongside everything else.

Security and access control

A knowledge assistant holds an index of your organisation's internal knowledge. That makes security a first-class concern, not a feature to bolt on later.

The most important security property is that the index is not a single searchable blob. It is partitioned by source, and each partition carries the access metadata from the original system. When a user queries the assistant, the retrieval layer filters by their identity before searching, so the index itself never leaks content across permission boundaries.

Network and data security works the same way as any internal service: the index and the model are either inside your network or behind your cloud tenancy's security perimeter. Traffic between the assistant and your source systems uses standard OAuth or service account credentials with scopes limited to read access. The assistant writes nothing back to your source systems.

For regulated environments, we align the data path to your compliance requirements. That typically means documenting exactly which data flows where, agreeing a retention policy for any query logs, and running the index on infrastructure that satisfies your data residency and audit obligations. GDPR, SOC 2 and ISO 27001 requirements are scoped per engagement rather than handled with a generic checkbox.

What does not work

• Indexing everything without a scope. The first instinct is to connect every source at once. A better approach is to start with the sources most likely to answer the questions your team asks most often, prove accuracy on those, then expand. Indexing low-quality or outdated content alongside high-quality content degrades answer accuracy.
• Assuming retrieval accuracy equals answer accuracy. A model can retrieve the right passage and still produce a misleading answer. Measure both independently rather than assuming one implies the other.
• Using it for time-sensitive decisions without a freshness check. The assistant answers from the index. If the index is a few hours old and a policy changed this morning, the answer will reflect the old policy. For decisions that depend on real-time data, wire in a freshness check or a human confirmation step.
• Skipping permission validation in the pilot. Permission errors in retrieval are subtle: the answer might look plausible even when it is composed from content the user should not have seen. Test permission boundaries explicitly during the pilot, not after launch.
• Treating a high no-answer rate as a failure. An assistant that declines to answer questions outside its knowledge is doing its job correctly. The goal is accurate answers on the questions the index covers, not maximum coverage at the cost of hallucination.

The engagement model

The shape of the work: a free pilot on one or two of your existing sources to measure accuracy on real questions, a fixed scope build that connects all your sources, wires in your permission model, and surfaces the assistant in Slack, Teams or a web widget, then ongoing operation to keep the index accurate and the answers current as your knowledge changes.

The build is software you own and run. There is no platform licence sitting in the middle. The model can be a hosted cloud model on your own API key or an open-weights model running on your infrastructure. The index is a vector database inside your environment. You own the data path end to end.

Frequently asked questions

---

Time to find an answer: before vs after a knowledge assistant

That is the full engine. The next step is a free pilot on your own knowledge base: real accuracy numbers, the hours quantified, and no obligation.

Get a free pilot