n8n queue mode on Azure Container Apps

2023-04-23 · Ben Williamson

Why I finally wrote this down

A single-container n8n instance is fine until long Magento or Meilisearch syncs pin the process. The UI freezes. Webhooks time out. Retries stack on the same overwhelmed box. I stood up a playground on Azure Container Apps to prove queue mode before asking for production-shaped networking.

Benchmarks also show multi-instance layouts without a queue. I still wanted the Bull/Redis path so workers could fail and retry without taking down the editor.

Architecture sketch

Four roles, cleanly separated:

  • Main: UI, workflow activation, scheduling. Keep webhook processing off this process when you have a dedicated webhook app.

  • Webhook receivers: accept inbound HTTP and enqueue work. Point WEBHOOK_URL here.

  • Workers: pull jobs from Redis and run executions. No public ingress required.

  • Shared state: Postgres for n8n metadata, Redis for the Bull queue.

Diagram content

This post includes an interactive diagram block. Open the original note in the primary docs app for full editing controls.

Separating scheduling from execution was the unlock. The editor stayed responsive while workers absorbed sync spikes.

Azure Container Apps layout

Same n8n image, multiple container apps (or revisions) differing by env:

  • Deploy Azure Cache for Redis reachable from every role. Private endpoints are the production default; a playground can start public and document the gap.

  • Shared Postgres with SSL settings that match your Flexible Server policy.

  • EXECUTIONS_MODE=queue on main, webhooks, and workers.

  • Shared N8N_ENCRYPTION_KEY across all roles so credentials decrypt everywhere.

  • Scale workers on CPU or queue depth. Keep main at a low min replica count.

Worker pods do not need HTTP ingress. Health probes that expect an open port will flap forever if you force ingress onto a worker that only talks to Redis and Postgres. Disable ingress on workers and watch the logs calm down.

Operational wins

  • Retry failed executions without wedging the whole instance.

  • Isolate community node installs to worker images when a node needs native deps.

  • Observe queue lag as an SLO instead of discovering OOM kills after the fact.

  • Prune execution data with EXECUTIONS_DATA_PRUNE and age/count caps so Postgres does not become the silent bottleneck.

Trade-offs

  • More moving parts: Redis ACLs, worker version skew, webhook routing that must hit the webhook app.

  • Build pipeline complexity if you want ACR tasks from a Dockerfile instead of pulling n8nio/n8n directly.

  • Encryption key and DB password rotation now touch every role at once.

Stay single-process when workflows are light and the editor never waits. Graduate to queue mode when concurrency or memory walls show up in production-shaped load, not when the architecture diagram looks lonely.

Failure modes I keep debugging

  • Workers with ingress health checks: probes fail because workers are not HTTP servers.

  • Webhook URL pointed at main: traffic bypasses the webhook role you sized for bursts.

  • Mismatched encryption keys: credentials decrypt on main and fail on workers.

  • Stale bad replicas: delete and recreate early when a first worker revision never becomes healthy.

Quick review before promote

  • Do webhooks hit the webhook app, not main?

  • Do workers run without public ingress?

  • Is N8N_ENCRYPTION_KEY identical across roles?

  • Is the Redis and Postgres contract written down for the next operator?