Field Feedback Intelligence & RAG Analytics Platform
Turns code-mixed voice notes from the field into governed, queryable business intelligence — with every model running on local GPU.
- Role
- Architect, Lead Engineer
- When
- 2025 – present
- Where
- Connexink (via Xtreme Thoughts)
- Client
- Global consumer-health company
- LLM calls for statistical questions
- LLM calls per qualitative answer
- near-duplicate taxonomy proposals
- faster vector queries after IPv6 fix
- independent RBAC layers
The problem
The problem
Field surveyors visit pharmacies, doctors and distributors and submit feedback as free text and multi-speaker Hindi/English voice recordings. Managers across a five-tier sales hierarchy need structured, trustworthy answers, and client data must never leave the network.
The system, drawn
The system, drawn
- 01
Two questions arrive
A manager asks something countable and something human. Most assistants send both to a language model. This one reads the question first.
- 02
Numbers never touch the model
Statistical, trend and location questions are routed to SQL over pre-computed TimescaleDB rollups. The answer is exact, instant and costs zero LLM calls.
- 03
Two retrievers, one ranked list
Qualitative questions run hybrid retrieval: vector search and Postgres full-text search in parallel, fused with Reciprocal Rank Fusion. Access scope is injected into the vector query itself, so out-of-scope feedback is never retrieved.
- 04
A second opinion on relevance
A cross-encoder re-reads each candidate against the question and reorders them. What the embedding thought was close isn't always what's useful.
- 05
An answer with receipts
Generation is grounded in the top passages with inline citations, capped at two LLM calls per answer, and every query writes an append-only audit row.
Illustrative questions · real mechanism
How it works · 7 parts
- 01
Ingestion pipeline
Idempotent, retriable Celery chain — each stage clears the output it owns before rewriting, so a retry from any point is safe. ASR + speaker diarization produce timestamped turns; anonymous speakers are mapped to business roles by LLM with heuristic fallback. Diarization degrades to single-speaker rather than failing.
- 02
Entity resolution
Three-stage canonicaliser: embedding ANN auto-resolves high-confidence matches, a cross-encoder reranks the shortlist, an LLM-as-judge decides the rest or declares the mention novel. A composite confidence score routes uncertain cases to human review; every resolution learns the surface form as an alias.
- 03
Governed taxonomy
Three-level concept tree with threshold-gated hard tags plus soft per-node similarity scores, aspect-level sentiment anchored to nodes, and a deterministic LLM-free quality score.
- 04
RAG with question routing
Statistical, trend and location questions answer from SQL over TimescaleDB rollups at zero LLM calls. Qualitative questions run hybrid retrieval (vector ANN + Postgres FTS fused by Reciprocal Rank Fusion), cross-encoder rerank and grounded generation with inline citations — capped at two LLM calls, with Redis-backed conversation memory.
- 05
Defence-in-depth RBAC
API scope dependencies, Postgres row-level security via per-session GUC (the database filters even if the app forgets), and scope filters injected into the vector query rather than post-filtered. Microsoft Entra SSO (OIDC + PKCE, JWKS validation) → short-lived session JWT; append-only audit row on every read and AI query.
- 06
Self-improving vocabulary
Nightly HDBSCAN over novel mentions raises drift events; schema-constrained LLM induction proposes taxonomy nodes behind four guardrails; chat telemetry is mined for aliases; nightly re-tagging applies approved vocabulary retroactively while versioning keeps history reproducible.
- 07
Model serving & reliability
Single provider protocol — business logic never imports a vendor SDK; Azure is a config change. GPU profile presets pick ASR/embedding/reranker/LLM per hardware tier. ASR runs in an isolated subprocess.
Stack
- Python
- FastAPI
- Celery
- SQLAlchemy 2.0 (async)
- PostgreSQL
- TimescaleDB
- Qdrant
- Redis
- MinIO
- faster-whisper
- pyannote
- BGE-M3
- Cross-encoder reranker
- vLLM
- Ollama (Qwen)
- HDBSCAN
- Microsoft Entra ID
- Docker