LLM Evaluation and Deployment Process (LLMOps)

How a team changes a prompt or model safely: run the evaluation suite, compare against the current version, release to a slice of traffic, watch the numbers and roll back fast if needed.

4 teams12 steps3 decisionsFree download

Edit this flowarrow_forward
LLM evaluation and release processAI engineer3Eval pipeline2Reviewer2Production5YesNoRe-runYesNoYesNoPrompt ormodelchange…Open a changewith the newversion3Run theevaluationsuite?Better orequal on everyslice?7Improve thechange5Review resultsand samples?Approverelease?8Release to 5%of trafficWatch livequality andcost?Healthy?RolledbackReleasedto all

Free to download. Edit it in Flow Studio to change the teams and steps, ask AI for changes, or export to PowerPoint, Word or draw.io.

Step by step

The same flow as a numbered list, ready for an SOP, a BRD or a ticket.

#Step
1Prompt or model change proposedStartAI engineer · step 2
2Open a change with the new versionDocumentPull request with prompt diff or model id, plus the reasonAI engineer · step 3
3Run the evaluation suiteGolden set + past-incident set + adversarial set; current vs candidateEval pipeline · step 4
4Better or equal on every slice?DecisionEval pipeline · Yes: step 5, No: step 7
5Review results and samplesRead 20 side-by-side outputs, check cost and latency deltasReviewer · step 6
6Approve release?DecisionReviewer · Yes: step 8, No: step 7
7Improve the changeAI engineer · Re-run: step 3
8Release to 5% of trafficConfig flag; same users stay on one version for clean comparisonProduction · step 9
9Watch live quality and costWait24-72 h: sampled judge score, feedback, errors, cost per requestProduction · step 10
10Healthy?DecisionProduction · Yes: step 12, No: step 11
11Rolled backEndProduction · Done
12Released to allEndProduction · Done

When to use it

Use it as the release gate for any AI feature in production, whether you're switching models in Azure AI Foundry, Amazon Bedrock or Vertex AI, or just editing a prompt.

Who's involved

  • AI engineer3
  • Eval pipeline2
  • Reviewer2
  • Production5

Yours works differently?

Describe your own process in a few sentences and Flow Studio draws it, with your teams and your rules.

Describe your processarrow_forward

For the technical team

The build decisions behind this flow. Each step above also carries its technical detail.

  • 01Eval set in the repo, split into slices (common, edge, adversarial, past incidents); run in CI on every prompt or model change.
  • 02Scorers: exact/regex checks, JSON schema validation, LLM-as-judge with a rubric, toxicity and PII detectors.
  • 03Release: prompt and model ids as config behind a flag; canary at 5% then 25% then 100%.
  • 04Online metrics: sampled judge score, user feedback rate, refusal rate, p95 latency, cost per request.
  • 05Rollback: flip the config to the last known good version; alert if online score drops below threshold for 30 minutes.

How to get it right

  1. 1Treat prompt changes like code changes: version them, review them and test them before release.
  2. 2Compare every candidate against the current version on the same examples. Absolute scores hide regressions.
  3. 3Keep a 'never again' set of past failures. A change that breaks one of them doesn't ship.
  4. 4Make rollback a single switch, and practise it.

Questions

What is LLMOps?

The practices for running language-model features in production: versioning prompts and models, automated evaluation, staged rollout, monitoring quality and cost, and fast rollback.

How do you evaluate an LLM before release?

Run a fixed test set of real cases through the current and the candidate version, score them with automatic checks (format, facts, safety) and an LLM judge calibrated against human ratings, and require the candidate to be at least as good on every critical slice.

More process flow templates

See all 34 process flow templates