Grounded Support Assistant
Local AI · Support engineering
The project in plain English
Why it matters
For Support and operations teams evaluating AI assistance
The problem
A confident-sounding AI answer is not enough for support work. Operators need sources, clear limits, and control over actions.
My contribution
The workbench searches support documents, shows cited answers, asks for approval before allowed tools run, and recommends a human when evidence is weak.
The value
Demonstrates an inspectable AI workflow where an operator can see the evidence and retain control over the next step.
- Applied AI
- Human approval
- System design
Scope & limits. A local engineering demonstration using a small fictional document set, not a deployed support system. It has no production identity, tenant isolation, or durable audit storage. Citation labels are checked; that does not prove every statement is correct.
Find evidence.
Cite sources.
Know when
to escalate.
A local-first support workbench with hybrid retrieval, cited answers, human-approved tools, and escalation when evidence is weak.
./run-demo.shAn engineering demonstration with fictional support documents. The workbench runs locally; it is not a production support service.
The workbench
Why this project
Most RAG demos stop at a chat box. Support work needs to know whether a passage supports an answer, whether a tool is safe to run, and when uncertainty belongs with a human. A fluent answer can still be wrong, and a real citation may not support its claim.
What it does
Keep evidence traceable
Heading-aware overlapping chunks retain filename, heading, ID, and timestamp. Content hashes skip duplicate documents.
Meaning and exact terms
Sentence Transformers and Chroma handle semantic search. BM25 finds headers, versions, and error codes. Normalized scores are fused 55/45 and deduplicated.
Stream with sources
FastAPI streams local Ollama answers over SSE. The server assigns citation labels from retrieved chunks and removes unsupported labels.
Make uncertainty visible
Retrieval strength, agreement, support count, and out-of-corpus signals form an explainable heuristic. Weak evidence recommends a human handoff.
Ask before execution
Three allowlisted local tools use strict Pydantic schemas and single-use proposal IDs. The model has no direct execution authority.
Inspect each decision
Local redacted events expose questions, retrieved chunks, confidence, proposals, approval decisions, results, and escalation.
An optional local cross-encoder reranks candidates. If it is unavailable, hybrid retrieval continues. If Ollama is unavailable, the app reports it explicitly instead of inventing support guidance.
Diagrams
flowchart TB Operator["Support operator"] --> UI["React + TypeScript workbench"] UI -->|"HTTP + streaming SSE"| API["FastAPI orchestration"] API --> Retriever["Hybrid retriever"] Retriever --> Chroma["ChromaDB vectors"] Retriever --> BM25["BM25 exact-term index"] API --> Reranker["Optional local cross-encoder"] API --> Ollama["Local Ollama model"] API --> Policy["Tool policy + human approval"] Policy --> Tools["Deterministic local tools"] API --> Audit["Process-local audit trail"]
Human-approved tools
Check a support matrix
Compare a supplied SDK version with a fictional, deterministic support matrix.
Inspect a propagation pair
Check supplied sentry-trace and baggage headers for the expected pair.
Prepare a human handoff
Package the question, verified findings, and requested next diagnostic step.
These functions have no network, filesystem mutation, production, or destructive capability. An operator reviews the proposed operation, arguments, and reason before approving or rejecting it.
Evaluation evidence
The repository includes a rerunnable, six-case fictional evaluation set. Its checked-in report records the following local results.
| Check | Result | Scope |
|---|---|---|
| Retrieval | Hit@1 / Hit@3: 1.000 | Semantic, BM25, and fused retrieval; 5 labelled cases over a 17-chunk fictional corpus. |
| Citation boundary | 10/10 checks passed | Controlled numeric-label filtering; 0 unsupported labels rendered. Does not measure claim-level entailment. |
| Tools + escalation | 3/3 expected outcomes | Denied execution, destructive refusal, and low-confidence escalation. |
| Local streaming | 309.473 ms first token 2529.735 ms completion | Medians across 3 warm FastAPI/Ollama SSE samples, qwen2.5:3b, reranking disabled. |
These are checked-in measurements, not a new benchmark run. The corpus is deliberately small and fictional. Warm-process timings do not represent clean startup, production traffic, hosted models, or external users.
Read the full report and reproduction details ↗Quick start
Install Miniforge or Conda and Ollama first. Allow approximately 3 GB for models and dependencies. Initial ingestion downloads the local embedding model.
git clone https://github.com/anujraja/grounded-support-assistant.git
cd grounded-support-assistant
cp .env.example .env
conda env create -f environment.yml
conda run -n grounded-support-assistant npm --prefix frontend ci
ollama pull qwen2.5:3b
./run-demo.shOpen localhost:5173 for the workbench. The API runs on localhost:8010, with interactive API documentation at localhost:8010/docs.
- Ingest the sample documents, then ask “What does the sentry-trace header do?” Inspect the streamed answer and its source chunks.
- Open Retrieval to compare vector, BM25, fused scores, and final ranks.
- Ask “Is JavaScript SDK version 99.0 supported?” Review and approve the proposed local version check, then inspect Audit.
- Try an out-of-corpus question to see escalation. Ask to delete a production project to see destructive-action refusal.
Boundaries and next steps
- Sample support documents and SDK matrices are fictional, not official Sentry documentation.
- Confidence is an explainable heuristic, not a calibrated probability. Citation labels constrain provenance, without verifying every claim.
- Proposals, approvals, and audit events are process-local state. There is no authentication, tenant isolation, rate limiting, or durable audit storage.
Production hardening would add identity and authorization, durable append-only records, larger evaluation datasets, claim-level citation checks, calibrated abstention, observability, and staged tool execution.