--- title: Production workflow description: Manage Memrail policy as reviewed JSONL, verify deployed scope and lifecycle, evaluate without live interference, and control application rollout and rollback. eyebrow: OPERATIONS keywords: Memrail production deployment, EMU JSONL workflow, agent policy rollout last_updated: 2026-09-10 --- # Production workflow Manage EMUs as reviewed policy-as-code: pull, edit, plan, apply, and commit the updated lock file. Test in staging, control lifecycle transitions explicitly, and verify deployed policy before enabling execution. The examples below use a separate `staging` workspace. Evaluate there before repeating a reviewed workflow against production with its own directory and lock file. Do not change a directory's workspace/project target casually: the lock file belongs to that deployed scope. ## Repository layout ```text your-project/ ├── emus/ │ ├── emus.jsonl │ └── .emu.lock.jsonl ├── src/ │ └── control/ │ ├── atoms.py │ ├── decision_points.py │ └── tools.py └── .github/ └── workflows/ └── memrail.yml ``` Commit both JSONL files. `emus.jsonl` is the desired policy; `.emu.lock.jsonl` tracks the deployed state and content hashes used for reliable plans. ## Pull remote state ```bash memrail emu-pull ./emus/ -w staging -p support-agent ``` Pull before editing when teammates or automation may have changed remote state. Do not hand-edit `.emu.lock.jsonl`. ## Edit one EMU per line ```jsonl {"emu_key":"support.enterprise_context","decision_point":"support.response","trigger":"state.customer.tier == 'enterprise'","action":{"type":"context_directive","directive":"Use the enterprise support protocol."},"policy":{"mode":"auto","priority":7},"expected_utility":0.75,"confidence":0.95,"state":"shadow"} ``` Formatting each policy on one physical line makes additions, changes, and removals unambiguous in code review. Use an editor or formatter that validates each line independently. Invoke the JSONL policy at `decision_point="support.response"`; keep routing names in trusted application code. Follow the shared [binding and lifecycle contract](/concepts/#bindings-and-lifecycle). ## Plan with validation ```bash memrail emu-plan ./emus/ -w staging -p support-agent --strict ``` Review: - policies to create, update, or archive; - trigger syntax and schema errors; - intended decision-point bindings and any explicit lifecycle transitions; - changes to action tools, versions, or interpolated arguments; - policy controls such as priority, cooldown, idempotency, and exclusions. Removing a tracked policy line represents archival on apply. Treat an unexpected archive as a blocking plan error. `--strict` validates the complete candidate set before writes and stops for errors, unavailable validation, or incomplete reports. `--validate` requests the same diagnostics without making them a blocking gate; `--validate-only` performs local JSONL/DSL checks and cannot be combined with `--strict`. ## Apply new policies in shadow ```bash memrail emu-apply ./emus/ \ -w staging \ -p support-agent \ --yes \ --strict \ --target-state shadow ``` Use shadow traces to compare proposed behavior under the [binding and lifecycle contract](/concepts/#bindings-and-lifecycle). Strict preflight is not a multi-record transaction. Serialize deployments, inspect apply results, and reconcile partial failures. The lock file records sync state; it is not a distributed lock. ## Validate the whole project ```bash memrail emu-validate -w staging -p support-agent memrail action-connectivity -w staging -p support-agent memrail tool-get-schema -w staging -p support-agent ``` Validation is broader than syntax. Investigate never-seen atoms, stale producers, mismatched types, placeholder gaps, and missing tools. The atom schema registry (ASR) observations may span team workspaces; they do not prove that an atom is present on every caller's path. Tool-schema checks do not prove a real handler, compatible version, or external credentials exist. Resolve `WARN-POLICY-GAP` findings with structured cooldown and `policy.idempotency` settings. Use literal state keys such as `"scope": ["refund.request_id"]`, not templates or the legacy `idempotency_key_template` field. ## Register tools coherently A Python tool registry used by CLI discovery belongs at module scope and should import without the rest of the application: ```python from memrail.tools import ToolRegistry registry = ToolRegistry() @registry.tool( name="ticket_escalator", description="Move a ticket to a named support queue", schema={ "type": "object", "properties": { "ticket_id": {"type": "string"}, "queue": {"type": "string"}, }, "required": ["ticket_id", "queue"], }, projects=["support-agent"], ) async def ticket_escalator(ticket_id: str, queue: str) -> dict: ... ``` ```bash memrail tool-list --path ./src/control --project support-agent memrail tool-register --file ./src/control/tools.py \ -w staging --project support-agent memrail action-connectivity -w staging -p support-agent ``` Replace the function body with your executor and keep its definition module importable by the CLI. Register a compatible schema in each project, then test the tool ID, version, arguments, permissions, and handler together. The server validator can fall back from project-level to workspace- or organization-level schemas. Runtime selection does not enforce tool availability or execute the tool. Maintain the project-level connection as an application contract; do not mistake a clean schema report for execution authorization. ## Observe before promotion During isolated shadow evaluation, compare candidate trace reasons, scores, and suppressions with actual application behavior. A matching shadow candidate has a `shadow_state` suppression and `passed: false`; it does not appear in `selected`. Sample: - expected positive matches; - expected negative cases; - missing optional and required context; - each model tag value, including `unknown`; - cooldown, exclusion, and idempotency suppression; - alternate callers of the same executor; - Memrail timeout and service failure; - executor success, retry, and terminal failure. Record mismatches as atom-contract, trigger, topology, or action-connectivity defects. Do not tune a trigger around missing instrumentation without correcting the producer. If you use `dry_run` against active/canary policies, the response can contain selections. Combined SDK helpers skip execution and ACK; custom dispatchers must also honor evaluation-only mode. Diagnostics and accounting may still occur. Reusing an invocation idempotency key with changed context, scope, or options returns a conflict; use a new key for a live request following a dry run. The SDK executor enforces returned policy and lifecycle metadata. Advisory tool calls/routes are not dispatched; advisory prompts/directives may run. `require_human` needs application-verified consent for every action type. Your handlers still enforce business permissions and duplicate protection. ## Promote through canary Configure and test an explicit application cohort decision before changing a policy to `canary`. The SDK executor skips canaries outside that cohort; the API does not assign a traffic percentage. Use stable entity-based assignment when exposure must be sticky. A skipped canary does not consume its action lock or cooldown. Successful execution ACK establishes those controls, so business-level duplicate protection must be in place before dispatch. After those controls and your review criteria are in place, use explicit lifecycle commands, then verify the remote result: ```bash memrail change-state support.enterprise_context canary \ -w staging -p support-agent memrail change-state support.enterprise_context active \ -w staging -p support-agent ``` These commands target staging. Production promotion is a separate reviewed operation in the production scope. Criteria should include match accuracy, action success, latency, review volume, and safe-fallback use—not only error rate. Do not rely on a JSONL state edit to perform the lifecycle change. ## Acknowledge and emit outcomes Apply the [outcome and acknowledgment contract](/concepts/#outcomes-and-acknowledgment). For an ACK-gated cooldown, `status="acknowledged"` starts the cooldown; `status="failed"` does not. This closes the loop: ```text selected → executed → acknowledged → event emitted → later policy can query outcome ``` ## CI planning example This example prepares a strict plan; it does not deploy. Set `MEMRAIL_REQUIREMENT` to your project's tested dependency requirement. A production apply job repeats strict validation and requires business tests, reviewed scope/lifecycle operations, and deployment approval. ```yaml name: Memrail policy plan on: pull_request: paths: ['emus/**'] workflow_dispatch: jobs: policy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: '3.12' - run: python -m pip install "${MEMRAIL_REQUIREMENT:?Set a tested Memrail build requirement}" env: MEMRAIL_REQUIREMENT: ${{ vars.MEMRAIL_REQUIREMENT }} - name: Plan and validate run: memrail emu-plan ./emus/ --strict env: AMI_API_KEY: ${{ secrets.AMI_API_KEY }} AMI_ORG: acme AMI_WORKSPACE: staging AMI_PROJECT: support-agent ``` Use protected environments or equivalent approval for consequential changes. Provide credentials only to trusted workflows; forked pull requests should not receive them. Do not print plans containing secrets or sensitive interpolated data. ## Rollback Rollback combines reviewed policy content with explicit scope and lifecycle verification: 1. revert the reviewed `emus.jsonl` commit; 2. run `emu-plan` and inspect the resulting content changes; 3. apply, inspect validation feedback, and verify remote definitions; 4. restore lifecycle through an explicit transition and reconcile the JSONL snapshot; 5. confirm the safe application fallback remains in effect; 6. investigate traces and executor outcomes before retrying promotion. Reverting source does not rewind remote history or undo external side effects. Review routing changes and duplicate protection under the [integration invariants](/concepts/#integration-invariants) before rollback. For immediate containment, change the affected EMU to `inactive` if your incident procedure authorizes it. Preserve traces and outcome events needed for analysis. ## Release gate - Plan shows only intended create, update, and archive operations. - JSONL and lock file are committed at the deployed revision. - Remote lifecycle and decision-point binding match the reviewed execution contract. - Strict candidate validation succeeds before writes; business and execution tests pass. - All trigger dependencies are present and type-compatible. - All actions are connected in the same project. - Side effects have deliberately scoped cooldown, literal-key idempotency, and bounded retry. - Shadow evidence covers intended and unintended matches without interfering with live arbitration. - Advisory/approval handling and evaluation-only dispatch suppression are explicit and tested. - A real canary cohort gate, metrics, and fallback use meet explicit thresholds. - ACK delivery failures have a reconciliation path independent of trace persistence. - Rollback has been exercised or is mechanically verifiable.