LLM Evaluation and Deployment Process (LLMOps)
How a team changes a prompt or model safely: run the evaluation suite, compare against the current version, release to a slice of traffic, watch the numbers and roll back fast if needed.
4 teams12 steps3 decisionsFree download
Free to download. Edit it in Flow Studio to change the teams and steps, ask AI for changes, or export to PowerPoint, Word or draw.io.
Step by step
The same flow as a numbered list, ready for an SOP, a BRD or a ticket.
| # | Step | Who | Next |
|---|---|---|---|
| 1 | Prompt or model change proposedStartAI engineer · step 2 | AI engineer | step 2 |
| 2 | Open a change with the new versionDocumentPull request with prompt diff or model id, plus the reasonAI engineer · step 3 | AI engineer | step 3 |
| 3 | Run the evaluation suiteGolden set + past-incident set + adversarial set; current vs candidateEval pipeline · step 4 | Eval pipeline | step 4 |
| 4 | Better or equal on every slice?DecisionEval pipeline · Yes: step 5, No: step 7 | Eval pipeline | Yes: step 5, No: step 7 |
| 5 | Review results and samplesRead 20 side-by-side outputs, check cost and latency deltasReviewer · step 6 | Reviewer | step 6 |
| 6 | Approve release?DecisionReviewer · Yes: step 8, No: step 7 | Reviewer | Yes: step 8, No: step 7 |
| 7 | Improve the changeAI engineer · Re-run: step 3 | AI engineer | Re-run: step 3 |
| 8 | Release to 5% of trafficConfig flag; same users stay on one version for clean comparisonProduction · step 9 | Production | step 9 |
| 9 | Watch live quality and costWait24-72 h: sampled judge score, feedback, errors, cost per requestProduction · step 10 | Production | step 10 |
| 10 | Healthy?DecisionProduction · Yes: step 12, No: step 11 | Production | Yes: step 12, No: step 11 |
| 11 | Rolled backEndProduction · Done | Production | Done |
| 12 | Released to allEndProduction · Done | Production | Done |
When to use it
Use it as the release gate for any AI feature in production, whether you're switching models in Azure AI Foundry, Amazon Bedrock or Vertex AI, or just editing a prompt.
Who's involved
- AI engineer3
- Eval pipeline2
- Reviewer2
- Production5
Yours works differently?
Describe your own process in a few sentences and Flow Studio draws it, with your teams and your rules.
Describe your processarrow_forwardFor the technical team
The build decisions behind this flow. Each step above also carries its technical detail.
- 01Eval set in the repo, split into slices (common, edge, adversarial, past incidents); run in CI on every prompt or model change.
- 02Scorers: exact/regex checks, JSON schema validation, LLM-as-judge with a rubric, toxicity and PII detectors.
- 03Release: prompt and model ids as config behind a flag; canary at 5% then 25% then 100%.
- 04Online metrics: sampled judge score, user feedback rate, refusal rate, p95 latency, cost per request.
- 05Rollback: flip the config to the last known good version; alert if online score drops below threshold for 30 minutes.
How to get it right
- 1Treat prompt changes like code changes: version them, review them and test them before release.
- 2Compare every candidate against the current version on the same examples. Absolute scores hide regressions.
- 3Keep a 'never again' set of past failures. A change that breaks one of them doesn't ship.
- 4Make rollback a single switch, and practise it.
Questions
What is LLMOps?
The practices for running language-model features in production: versioning prompts and models, automated evaluation, staged rollout, monitoring quality and cost, and fast rollback.
How do you evaluate an LLM before release?
Run a fixed test set of real cases through the current and the candidate version, score them with automatic checks (format, facts, safety) and an LLM judge calibrated against human ratings, and require the candidate to be at least as good on every critical slice.
More process flow templates
- AI feature integrationTaking an AI feature from idea to production: pick the use case, test models against real examples, add guardrails, pass a security review, and roll out in stages with monitoring.
- RAG pipelineHow retrieval-augmented generation answers questions from your own documents: documents are split, embedded and indexed, then each question pulls the best passages and the model answers with citations.
- AI agent with tools and MCPHow an agentic AI system works through a task: it plans, calls tools through MCP servers with scoped access, asks a person before risky actions, checks results and loops until the goal is met.
- MCP server integrationConnecting an internal system to AI assistants through a Model Context Protocol server: design the tools, secure them, test them, review them and roll them out with monitoring.