All case studies

Case study

Macoy: a RAG assistant that says "I don't know"

A bilingual assistant for a credit cooperative that answers from the coop's own documents with hybrid search, Claude reranking, grounding checks, and role-scoped retrieval.

Client
Manatal Coop
Period
March 2026 – present
Role
Designed and built the assistant end to end
  • Django REST Framework
  • PostgreSQL + pgvector
  • sentence-transformers
  • Claude API
  • Celery
  • Redis
  • Docker

Manatal Coop's members ask the same questions every day: loan requirements, savings rules, policies, and figures from the annual report. The answers already existed in the coop's published content. Macoy is a bilingual (English and Tagalog) assistant that answers from that content, cites its sources, and declines to answer when the sources don't support one.

It runs in two places from one backend. A public widget on manatalcoop.com answers from public content only. The member app answers from the internal documents the member's role allows, and can look up the logged-in member's own account balances.

How content gets in

Ingestion: staff publish content, a Celery job chunks it, Claude vision reads images, unchanged content is skipped, and local embeddings are stored in pgvector with a full-text index

Publishing is the trigger, so staff never run an indexing step. Unpublishing removes the chunks. A content hash skips re-embedding when nothing changed.

How a question gets answered

Question flow: rewrite the question, apply the access scope, run vector and keyword search, merge with rank fusion, rerank with Claude, answer with citations, and verify every claim before replying

Decisions and why

  • Access control before ranking. The public or role scope is a SQL filter inside both searches. Content a user may not see never reaches ranking, the prompt, or the model.
  • Hybrid search. Vectors catch meaning; keywords catch exact terms like loan product names and peso amounts. Reciprocal rank fusion merges the two ranked lists without tuning score scales (weights 0.8 semantic, 0.2 keyword).
  • Local embeddings. No per-call cost, re-embedding everything is free, and member questions are not sent to an embedding API.
  • One paid rerank. Retrieval over-fetches with cheap SQL; Claude reranks 50 candidates once. Claude Haiku handles rewriting, reranking, and answering.
  • Abstain instead of guess. No context, or claims the sources don't support, produce a no-answer reply. Every answer stores grounding metadata and member ratings, and a report command lists weak answers for review.
  • Tools only where safe. The balance tool exists only in member sessions and only for the logged-in member. The public widget has no tools.
  • Consent first. Members accept AI data-sharing consent before their first message.

What I would add next

  • An offline evaluation set built from rated answers, run before any prompt or model change.
  • Routing by question type, so simple lookups skip the rerank call.
  • Context for each chunk generated by an LLM at ingest time, beyond today's fixed type, title, and category prefix.