RAG Pipeline Architecture Flowchart
How retrieval-augmented generation answers questions from your own documents: documents are split, embedded and indexed, then each question pulls the best passages and the model answers with citations.
5 teams14 steps1 decisionsFree download
Free to download. Edit it in Flow Studio to change the teams and steps, ask AI for changes, or export to PowerPoint, Word or draw.io.
Step by step
The same flow as a numbered list, ready for an SOP, a BRD or a ticket.
| # | Step | Who | Next |
|---|---|---|---|
| 1 | Documents added or changedStartSharePoint, Google Drive, Confluence, Zendesk, PDFs; change events or a nightly syncContent sources · step 3 | Content sources | step 3 |
| 2 | User asks a questionStartUser · step 4 | User | step 4 |
| 3 | Extract and clean the textParse PDF/HTML/DOCX, OCR scans, strip boilerplate, keep headings and tablesIngestion pipeline · step 5 | Ingestion pipeline | step 5 |
| 4 | Rewrite the question for searchResolve follow-ups using chat history; optional multi-query expansionLanguage model · step 8 | Language model | step 8 |
| 5 | Split into chunks300-800 tokens, 10-20% overlap, split on headings; attach source, date and access listIngestion pipeline · step 6 | Ingestion pipeline | step 6 |
| 6 | Create embeddingsEmbedding model (e.g. text-embedding-3, Cohere, bge); batch, cache by content hashIngestion pipeline · step 7 | Ingestion pipeline | step 7 |
| 7 | Store in the search indexDocumentpgvector, Azure AI Search, Pinecone or Weaviate; plus a keyword index for hybrid searchIngestion pipeline · Searched at query time: step 8 | Ingestion pipeline | Searched at query time: step 8 |
| 8 | Find matching passagesHybrid search, filter by the user's permissions, top 20-50 candidatesRetrieval service · step 9 | Retrieval service | step 9 |
| 9 | Rerank and keep the bestCross-encoder reranker to 5-8 passages; drop below a relevance thresholdRetrieval service · step 10 | Retrieval service | step 10 |
| 10 | Enough relevant content?DecisionRetrieval service · Yes: step 11, No: step 12 | Retrieval service | Yes: step 11, No: step 12 |
| 11 | Write the answer with citationsAnswer only from the passages; cite passage ids; structured outputLanguage model · step 13 | Language model | step 13 |
| 12 | Say it isn't coveredEndUser · Done | User | Done |
| 13 | Check the answer is groundedFaithfulness check: every claim maps to a cited passage; strip or flag unsupported claimsLanguage model · step 14 | Language model | step 14 |
| 14 | Answer with sourcesEndLinks to the source sections; thumbs up/down feeds the eval setUser · Done | User | Done |
When to use it
Use it to explain or build an AI assistant that answers from company knowledge (policies, product docs, contracts, tickets) instead of from the model's general memory.
Who's involved
- Content sources1
- Ingestion pipeline4
- Retrieval service3
- Language model3
- User3
Yours works differently?
Describe your own process in a few sentences and Flow Studio draws it, with your teams and your rules.
Describe your processarrow_forwardFor the technical team
The build decisions behind this flow. Each step above also carries its technical detail.
- 01Ingestion: connectors for each source, text extraction (incl. OCR for scans), cleaning, and a content hash so unchanged docs aren't re-embedded.
- 02Chunk metadata: source URL, title, section, updated date and access-control list, stored with each vector.
- 03Retrieval: hybrid BM25 + vector search, top 20-50 candidates, then a cross-encoder reranker down to 5-8 passages.
- 04Generation: system prompt that requires citations by passage id; refuse when retrieval confidence is low.
- 05Evaluation: track retrieval recall@k and answer faithfulness on a fixed question set on every change.
How to get it right
- 1Most bad answers come from bad retrieval, not the model. Check which passages were found before changing the prompt.
- 2Keep document permissions on every chunk and filter at query time, so people only get answers from what they're allowed to read.
- 3Ask the model to answer only from the passages and to say when they don't cover the question. Show the citations to users.
- 4Re-index on a schedule or on change. Stale chunks are a quiet source of wrong answers.
Questions
What is a RAG pipeline?
Retrieval-augmented generation finds the most relevant passages from your own documents for each question and gives them to the language model, so it answers from your content and can cite it.
What chunk size should I use for RAG?
A common starting point is 300 to 800 tokens with 10 to 20% overlap, split on headings and paragraphs rather than fixed lengths. Test on your own questions: dense legal text and short FAQs need different sizes.
Do I need a vector database?
You need somewhere to store embeddings and search them. Postgres with pgvector is enough for most teams; dedicated stores like Pinecone, Weaviate or Azure AI Search help at very large scale. Hybrid search (keywords plus vectors) usually beats vectors alone.
More process flow templates
- AI feature integrationTaking an AI feature from idea to production: pick the use case, test models against real examples, add guardrails, pass a security review, and roll out in stages with monitoring.
- AI agent with tools and MCPHow an agentic AI system works through a task: it plans, calls tools through MCP servers with scoped access, asks a person before risky actions, checks results and loops until the goal is met.
- MCP server integrationConnecting an internal system to AI assistants through a Model Context Protocol server: design the tools, secure them, test them, review them and roll them out with monitoring.
- LLM evaluation and releaseHow a team changes a prompt or model safely: run the evaluation suite, compare against the current version, release to a slice of traffic, watch the numbers and roll back fast if needed.