Is your knowledge base ready for RAG? A practical checklist

A useful knowledge assistant starts with a question the business can answer reliably. Before choosing a model or uploading a document library, check whether the underlying knowledge has clear owners, permissions and evidence.

Retrieval-augmented generation (RAG) brings retrieved information into a language model's answering process. That makes the quality and scope of retrieval part of the product design. Microsoft's RAG overview describes the pattern and the challenges around access, relevance and response time.

The checklist below is a suggested workshop approach from Kaizenix. Consider a hypothetical operations assistant answering: “Which return procedure applies to this region and product?” The example is illustrative, not a customer deployment or a claim of measured results.

1. Choose questions before documents

Ask the intended users for a small set of questions they regularly investigate. Record what a useful answer must contain, which decisions it supports and when a person should take over. “Find the policy” and “approve a refund” are different scopes; an initial knowledge assistant can help with the first without being authorized to perform the second.

For the returns example, write down the region, product, purchase date and policy version needed to answer. If users cannot specify what evidence would settle the question, start with that discussion before building an assistant.

2. Give each source an owner and a date

Create a source register with the document location, business owner, audience and review date. Identify duplicates, superseded versions, scanned documents and content that is missing altogether. Decide which system is authoritative when two documents disagree.

Agree what happens when a source changes or is deleted. A successful upload is a point-in-time event; keeping an assistant useful requires an ongoing route for approved changes to reach its retrieval layer. For the pilot, select a bounded collection whose owner can inspect every included document.

3. Preserve permissions through retrieval

List the user groups involved and the records each group may access. Test the boundary with representative accounts. The retrieval service should enforce the relevant access rules before restricted material can enter an answer's context; an instruction asking the model to keep a secret is not an access-control system.

Microsoft's security-filter documentation illustrates one implementation pattern. The right mechanism depends on the source system and identity model. Ask the implementation team to demonstrate the full path, including access changes and revoked permissions, rather than assuming document labels alone enforce authorization.

4. Define what an evidence-backed answer looks like

Agree a simple answer format: a direct response, links to the supporting source, the relevant version or date, and any uncertainty that requires review. Check that a reader can open the cited source and find the passage supporting the answer. A citation that points to a loosely related document does not establish that the answer follows from it.

When the evidence is missing or conflicting, specify a useful next step: ask for a missing detail, explain the conflict or route the question to the source owner. For the returns assistant, a missing purchase date should prompt clarification when that date changes the applicable policy.

5. Build a review set with difficult cases

Collect ordinary questions alongside incomplete, ambiguous and unanswerable ones. Include outdated policies, different access groups, similar product names and multiple languages if they are in scope. Beside each case, record the expected supporting document, acceptable answer characteristics and escalation behavior.

Review retrieved evidence separately from the generated answer so the team can distinguish a search problem from an answering problem. Google's Gen AI evaluation overview provides context on evaluating application responses. Choose criteria for your own workflow and keep the examples repeatable; a polished demonstration is not the same as an agreed acceptance test.

6. Map the exceptions and the handoff

Decide what happens if a connector fails, two policies conflict or the assistant cannot establish which source is current. Name the person who can pause the pilot, investigate a report and approve a correction. Give reviewers a way to flag an unsupported answer without copying restricted information into a public feedback channel.

Our companion guide, Before automating a business workflow, map the exceptions, offers a practical way to turn these conditions into review points and acceptance examples.

7. Finish with a small, assessable pilot

Write a one-page pilot brief containing the user group, permitted sources, supported questions, review set and operating owner. Start with read-only answers where that meets the business need. Agree who assesses the results and what evidence is needed before expanding the audience, source collection or allowed actions.

The outcome of this preparation might be a RAG pilot. It might also be a decision to repair the source library or improve ordinary search first. Either is a useful result when the team can explain the decision and the next step.

Take the next step with Kaizenix

Explore Kaizenix's RAG development services for source-linked enterprise search and knowledge assistants. The enterprise assistant use case shows the kinds of workflow questions to explore during discovery. To discuss a concrete task, contact the Kaizenix team with the question, the source owners and the constraints that matter.

Written by Kaizenix Team with AI assistance. Published on 7 November 2026. The checklist is educational guidance and uses a hypothetical example.