A documentation assistant for an operations team

Context

The client runs warehousing and distribution for several retail brands. Their operations team of around 60 people works from a large and growing body of documents: health and safety policies, standard operating procedures for each site, supplier contracts, customer service agreements and a decade of internal memos.

Finding the right answer was slow. Supervisors estimated they spent several hours a week searching shared drives or asking the one colleague who "just knows". New starters found it especially hard.

The challenge

An internal team had already built a chatbot prototype by connecting a language model to the document folder. In demos it looked impressive. In daily use, it was a problem:

• It gave confident but wrong answers, sometimes mixing up procedures from different sites.
• It couldn't say where an answer came from, so supervisors had no way to check it.
• Nobody could tell whether a change to the prompt made it better or worse overall.

The operations director was ready to abandon the idea. We were asked to find out whether a reliable assistant was achievable, and to build it if so.

Our approach

Building the test set first

We ran two workshops with supervisors from different sites and wrote 150 real questions together, each with the correct answer and the document it comes from. The questions covered easy lookups ("What is the maximum stack height in aisle racking at Site B?"), questions that need two documents, and questions that should be answered with "this isn't covered".

Scored against this set, the existing prototype answered 52% of questions correctly. That gave everyone, including the sceptics, a clear and shared starting point.

Getting retrieval right

Most of the prototype's mistakes came from retrieving the wrong passages, not from the language model itself. We rebuilt the ingestion pipeline:

• Documents are split along their natural structure (sections, numbered steps, clauses) instead of fixed-size chunks.
• Each passage is tagged with metadata: document type, site, effective date and owner.
• Questions that mention a site or document type filter results before ranking, which removed most cross-site mix-ups.
• Superseded versions of documents are excluded automatically.

Passages and embeddings are stored in PostgreSQL with pgvector, which the client already ran, so there was no new database to operate.

Answers that show their working

Every answer includes links to the passages it is based on, with the relevant text highlighted. The assistant is instructed to answer only from the retrieved passages, and to say clearly when the documents don't cover a question rather than guessing. We tested that behaviour explicitly with the "not covered" questions in the test set.

Improving against the test set

Each change to retrieval, prompts or model settings was scored against the full test set before it was kept. Over five iterations, accuracy climbed from 52% to 87%. Just as important, the rate of confident wrong answers fell sharply, because the assistant learned to decline when retrieval found nothing relevant.

How it works

1. New and updated documents are picked up nightly from the shared drive, split, tagged and indexed.
2. A supervisor asks a question in the team's existing chat tool.
3. The assistant retrieves the most relevant passages, filtered by site and document type where the question allows.
4. The language model writes an answer using only those passages and includes source links.
5. Users can rate an answer. Low-rated answers are reviewed weekly and, where useful, added to the test set.

Results

On the expert-written test set, 87% of answers were rated correct before rollout, up from 52% for the prototype. Every answer links to its source, so supervisors can check anything important in seconds.

After the first month of use, supervisors reported spending noticeably less time searching for procedures, and new starters said it was the first place they looked for answers.

What we delivered

• A retrieval-augmented assistant with source citations, integrated with the team's chat tool.
• An ingestion pipeline for new and updated documents, with metadata tagging.
• An evaluation test set and a repeatable quality report the team can run after any change.
• A feedback loop for collecting and reviewing low-rated answers.
• Documentation covering architecture, prompts, evaluation and day-to-day operation.

What we learned

The difference between a demo and a dependable tool was almost entirely in evaluation and retrieval, not in the choice of language model. A test set written by the users themselves turned a vague debate about whether "AI works" into a concrete, measurable project.