AdvisorDesk: AI-Powered Advisory Content Platform

Problem

Financial advisory firms produce a steady stream of written guidance, but their clients cannot easily get answers from it. The guidance lives in documents and portals, so a client with a question has to search through them or wait for an advisor to reply. A generic chatbot does not solve this either. In a regulated space, an answer that cannot be traced back to the firm's own published guidance creates risk instead of value.

Approach

I treated the firm's published content as the single source of truth and built the platform around it.

• An admin CMS lets advisors create, edit, tag, and publish guidance. Publishing is what makes content available to the assistant. Drafts stay invisible to it.
• A client portal includes a RAG assistant that answers questions using vector search over published content only. Every answer includes numbered citations that link back to the source articles. When nothing published covers the question, the assistant says so instead of guessing.
• An agentic content layer exposes the CMS operations for drafting, tagging, searching, and publishing as MCP tools, so a content manager can run the editorial workflow through natural language, either from an agent panel inside the CMS or from Claude.
• An evaluation loop measures how well answers are supported by their sources, flags the weak ones, and checks every proposed content fix before it is accepted.

Architecture & Tech

Two Next.js applications, the admin CMS and the client portal, share one FastAPI backend. The backend is a modular monolith, and its import boundaries are enforced in CI, so the layering holds as the code grows. TypeScript API types are generated from the backend's OpenAPI spec, and CI fails when the committed spec or the generated types drift, which catches frontend and backend mismatches before they ship.

Content and embeddings live in Supabase, which provides Postgres with pgvector. Publishing an article chunks it, embeds it, and writes the vectors in one transaction. Retrieval runs a similarity search over published chunks, OpenAI generates the cited answer, and the answer streams to the browser over Server-Sent Events.

All 14 MCP tools live in one registry with two ways in. The agent panel in the CMS calls them in process, and a remote MCP server exposes the same tools to Claude behind an OAuth 2.1 authorization flow. A tool behaves the same way no matter which side calls it. Advisors sign in with Google, and access is limited to an email allowlist.

The three services run as Docker images on a single AWS EC2 host behind Caddy, which handles TLS.

Evaluation

I evaluated the assistant on 80 questions across three runs, covering answerable questions, near misses, off topic questions, and questions that need more than one source. It reached 93% mean source support, and it refused correctly every time the published content had no answer.

The same harness drives a feedback loop. Questions that retrieve poorly are grouped into a weak query report for content managers. When someone proposes a content fix, the platform compares evaluation runs from before and after, and it accepts the fix only if no question got worse. In one rehearsal, a deliberately bad draft raised the overall score and was still blocked, because two individual questions regressed.

Results & Impact

• In production, the median time to first token is 733 ms, and the estimated model cost is about $0.0013 per answer.
• An automated check runs every six hours against the deployed site. It asks a real question and verifies that a cited answer streams back, so it tests the whole chat path and not only a health endpoint.
• The codebase has roughly 1,000 backend tests and 500 frontend tests, and CI enforces a 95% coverage floor on the backend.
• Content managers can draft, search, and publish articles from Claude through the OAuth protected MCP tools.

The live site is seeded with sample guidance for a fictional advisory firm. It holds no real client data. This project is also the case study for my talk, Beyond RAG: Building an Evaluated, Self-Improving Agentic AI System.

Try it

The client portal is public and needs no login, including the assistant. The admin CMS runs at admin.advisordesk.tyagiakanksha.com and accepts only allowlisted Google accounts, so it is described here instead of linked.

What I'd do next

I would finish a larger round of human labels to calibrate the evaluation judge, since the current agreement check rests on a small sample. I would also move the in process rate limit and latency counters into Redis, so the API can run more than one worker.