Next.js 16 Background Jobs and Queues with Redis

Next.js background jobs are units of slow, retryable, or resource-heavy server work that run outside the request that created them. The Next.js application validates a command, stores a small job in a durable queue, responds quickly, and lets an independently deployed worker process email, AI, media, import, or webhook work.

1. Blog #20Docker + TraefikPublished
2. Blog #21Compose StackPublished
3. Blog #22RedisPublished
4. Blog #23Jobs + QueuesCurrent
5. Blog #24MonitoringPublished

How Do Background Jobs Work in Next.js?

A Next.js background-job architecture sends slow or retryable work to a queue instead of completing it inside the HTTP request. The web application adds a job to a queue such as BullMQ backed by Redis, returns a response, and a separate worker processes the job asynchronously.

Why Next.js Apps Need Background Processing

HTTP requests have finite lifetimes. A browser can disconnect, a reverse proxy can time out, a serverless invocation can end, or a deployment can replace the process. Returning a response does not guarantee arbitrary in-process work will continue. A durable queue creates an explicit handoff, retry policy, concurrency limit, and observable lifecycle.

Next.js 16 also provides after(), which schedules code after a response or prerender finishes, and self-hosted next start waits for pending callbacks during graceful shutdown. That is useful for bounded post-response work such as logging. It is not a general durable queue: it does not replace persistent delivery, independent workers, retry/backoff state, workload smoothing, or queue observability.

Producer, Redis Queue, and Worker

The producer is trusted server-only Next.js code. BullMQ stores and coordinates jobs in Redis. A worker is a long-running Node process that imports the handler and listens to the exact same queue name. The browser never connects to Redis, and the worker normally exposes no public HTTP port.

Redis and BullMQ

Redis is infrastructure, not a complete job system by itself. BullMQ adds claiming, locks, retries, delayed jobs, concurrency, events, progress, and cleanup semantics. This guide uses BullMQ's Redis backend. Current BullMQ v6 also provides Redis client adapters and a PostgreSQL backend abstraction, so read the documentation for the exact major version you adopt rather than mixing v5 and v6 APIs.

Do not assume cache Redis and queue Redis have identical requirements. A cache may permit eviction and no persistence; important jobs may require noeviction, reviewed AOF/RDB persistence, replication, recovery tests, adequate memory, and a dedicated failure domain.

Install and Configure the Queue

In a real Next.js repository, pin reviewed versions using its existing package manager. The example below uses BullMQ's default Redis connection options; current BullMQ can also use supported Redis client adapters, whose reconnect and duplicate() behavior must meet worker requirements. Keep shared connection configuration in server/worker code, validate environment at startup, and never use a NEXT_PUBLIC_ queue credential.

npm install bullmq zod

import type { ConnectionOptions } from 'bullmq'

const redisUrl = process.env.REDIS_URL
if (!redisUrl) throw new Error('REDIS_URL is required')

const url = new URL(redisUrl)
const baseConnection: ConnectionOptions = {
  host: url.hostname,
  port: Number(url.port || 6379),
  username: url.username || undefined,
  password: url.password || undefined,
  tls: url.protocol === 'rediss:' ? {} : undefined,
}

export const producerConnection: ConnectionOptions = {
  ...baseConnection, maxRetriesPerRequest: 1,
}
export const workerConnection: ConnectionOptions = {
  ...baseConnection, maxRetriesPerRequest: null,
}

Create a Queue and Add Jobs

export const REPORT_QUEUE = 'reports-v1'

export type ReportJob = {
  schemaVersion: 1
  reportId: string
  accountId: string
}

import 'server-only'
import { Queue } from 'bullmq'
import { producerConnection } from './connection'
import { REPORT_QUEUE, type ReportJob } from './contracts'

export const reportQueue = new Queue<ReportJob>(REPORT_QUEUE, {
  connection: producerConnection,
  defaultJobOptions: {
    attempts: 4,
    backoff: { type: 'exponential', delay: 2_000 },
    removeOnComplete: { age: 86_400, count: 1_000 },
    removeOnFail: { age: 604_800, count: 5_000 },
  },
})

import { z } from 'zod'
import { reportQueue } from '@/lib/queue/report-queue'

const inputSchema = z.object({ reportId: z.string().uuid() }).strict()

export async function POST(request: Request) {
  const user = await requireUser(request)
  const parsed = inputSchema.safeParse(await request.json())
  if (!parsed.success) return Response.json({ error: 'Invalid request' }, { status: 400 })

  await requireReportAccess(user, parsed.data.reportId)
  const task = await createPendingTask(user.accountId, parsed.data.reportId)
  await reportQueue.add('generate-report', {
    schemaVersion: 1,
    reportId: parsed.data.reportId,
    accountId: user.accountId,
  }, { jobId: `report-${task.id}` })

  return Response.json({ taskId: task.publicId, status: 'queued' }, { status: 202 })
}

Design Safe Job Payloads

A job payload is durable data that may appear in Redis, dashboards, logs, backups, and failed-job tooling. Store stable identifiers and a schema version, then load authorized current data in the worker. Do not store passwords, tokens, raw authorization headers, provider API keys, full private documents, or large binary files. Put files in object storage and enqueue an opaque object identifier.

Create a Separate Worker

import { Worker } from 'bullmq'
import { z } from 'zod'
import { workerConnection } from '../lib/queue/connection'
import { REPORT_QUEUE } from '../lib/queue/contracts'

const jobSchema = z.object({
  schemaVersion: z.literal(1), reportId: z.string().uuid(), accountId: z.string().uuid(),
}).strict()

const worker = new Worker(REPORT_QUEUE, async (job) => {
  const payload = jobSchema.parse(job.data)
  const claimed = await claimTaskOnce(job.id, payload.accountId)
  if (!claimed) return { status: 'already-processed' as const }
  await job.updateProgress(20)
  const result = await generateReport(payload.reportId, payload.accountId)
  await saveReportAndCompleteTask(job.id, result)
  await job.updateProgress(100)
  return { status: 'completed' as const }
}, { connection: workerConnection, concurrency: 4 })

worker.on('failed', (job, error) => {
  console.error('Report job failed', { jobId: job?.id, name: error.name })
})

async function shutdown(signal: string) {
  console.info('Closing worker', { signal })
  await worker.close()
}
process.once('SIGTERM', () => void shutdown('SIGTERM'))
process.once('SIGINT', () => void shutdown('SIGINT'))

The handler validates the durable payload again because queued data is still an input boundary. The worker receives only the database and provider permissions it needs. Logs contain an internal job ID and error category, never connection URLs, tokens, payload contents, or sensitive provider responses.

Job States, Retries, and Backoff

A typical job moves from waiting to active and then completed. A transient failure can move it back to a delayed retry; a permanent error or exhausted attempt count ends in failed state. BullMQ retains or removes records according to explicit policy. “Exactly once” should not be assumed: distributed workers provide at-least-once behavior in practical failure cases.

Retry network timeouts, temporary rate limits, and recoverable provider failures. Do not blindly retry invalid payloads, revoked access, unsupported file formats, or hard quota failures. Honor Retry-After when supported, add jitter for large fleets, and cap both attempts and total time.

Job IDs and Idempotency

A deterministic job ID can suppress duplicate additions while the original job exists, but it does not make side effects idempotent. A worker may send an email and crash before completion is recorded. Use an application idempotency key, a unique database constraint or atomic state transition, and provider-supported idempotency where available. Define ordering explicitly; multiple workers and retries can reorder independent jobs.

Concurrency and Horizontal Scaling

BullMQ concurrency lets one worker process several asynchronous jobs. It helps I/O-bound work but does not create more CPU. CPU-heavy image or PDF work should use worker threads, child processes, or specialized services with conservative limits. Calculate database, Redis, memory, file-descriptor, and provider capacity before increasing replicas.

Email, AI, Media, and Webhook Patterns

Delayed and Scheduled Jobs

delay is useful for one future attempt, such as a short follow-up. BullMQ v6 removed the legacy repeat APIs, so repeated schedules use the current Job Scheduler API. Schedulers still need one logical owner, timezone and daylight-saving decisions, catch-up behavior, overlap prevention, and idempotency. Next.js or a host can trigger cron-like endpoints, but cron is a trigger—not a durable execution engine.

await reportQueue.upsertJobScheduler(
  'daily-account-reports-v1',
  { pattern: '0 15 3 * * *', tz: 'UTC' },
  {
    name: 'generate-scheduled-reports',
    data: { schemaVersion: 1, reportId, accountId },
    opts: { attempts: 4, backoff: { type: 'exponential', delay: 2_000 } },
  },
)

A Job Scheduler produces the next delayed job when the preceding one begins processing, so a busy queue or insufficient worker capacity can make runs less frequent than the nominal interval. Monitor scheduler lag instead of treating it as a wall-clock guarantee.

Expose Safe Job Status

Do not give clients raw BullMQ job IDs, stack traces, payloads, or internal failure reasons. Store an application-owned task row with an opaque public ID, account ownership, safe state, bounded progress, and a user-facing error category. An authorized Route Handler can return queued, running, completed, or failed. Polling is often sufficient; Server-Sent Events or WebSockets add connection and authorization work.

Database and Queue Consistency

A database commit and Redis enqueue are separate operations. If the write succeeds but enqueue fails, the application can lose required work. The transactional outbox pattern writes the business change and an outbox row in one PostgreSQL transaction. A dispatcher later publishes unsent rows, records delivery, and retries safely. The consumer still remains idempotent.

Monitoring, Failed Jobs, and Backpressure

Monitor queue depth, oldest waiting age, job duration percentiles, throughput, retry rate, exhausted jobs, stalled jobs, active concurrency, event-loop lag, Redis latency and memory, worker restarts, downstream errors, and business completion rate. Queue depth alone can look healthy while one old tenant job is stuck.

Retain failed jobs long enough to diagnose and replay safely, but cap count and age. Alert on sustained age, failure ratio, and stalled processing. A dead-letter queue is useful only with an owner, redrive procedure, payload retention policy, and protection against replaying permanent or unsafe failures.

Graceful Worker Shutdown

On SIGTERM, stop accepting new jobs, allow active handlers to finish within the platform grace period, close the BullMQ worker, then exit. Make Kubernetes, Compose, or the process manager grace period longer than normal job completion or design checkpoints for longer work. A forced kill can lead to another delivery, which is why idempotency remains mandatory.

Docker Production Architecture

Build the web app and worker from the same reviewed source and immutable image, then select a different command. The web service joins the public proxy network and private application/data networks. The worker joins only private networks. Redis and PostgreSQL publish no host ports. The worker publishes no port at all.

services:
  app:
    image: registry.example/app:${APP_IMAGE_TAG:?required}
    command: ["node", "server.js"]
    networks: [proxy, app_private, data_private]
  worker:
    image: registry.example/app:${APP_IMAGE_TAG:?required}
    command: ["node", "dist/workers/report-worker.js"]
    restart: unless-stopped
    stop_grace_period: 60s
    networks: [app_private, data_private]
    # No ports: worker is not a public service.
  redis:
    image: redis:7.4-alpine
    command: ["redis-server", "--appendonly", "yes", "--maxmemory-policy", "noeviction"]
    volumes: [redis_data:/data]
    networks: [data_private]
  postgres:
    image: postgres:17-alpine
    volumes: [postgres_data:/var/lib/postgresql/data]
    networks: [data_private]
networks:
  proxy: { external: true }
  app_private: { internal: true }
  data_private: { internal: true }
volumes:
  redis_data:
  postgres_data:

The example is not copied into a live stack because this repository has none. Pin image digests or approved versions, supply secrets through deployment configuration, add real health/readiness behavior, test backup and recovery, and compile worker TypeScript as part of the image build. Do not run TypeScript directly in production without an intentional supported runtime.

Production Security Checklist

• Keep REDIS_URL server-only and out of logs.
• Publish neither Redis nor worker ports.
• Authenticate and authorize every enqueue operation.
• Validate strict, versioned payloads at producer and worker.
• Keep secrets and sensitive content out of jobs.
• Use least-privilege database, storage, and provider credentials.
• Verify webhook signatures before trusting events.
• Redact provider errors returned to clients.
• Bound retries, job size, runtime, and concurrency.
• Enforce provider rate limits and cost budgets.
• Review Redis persistence and maxmemory policy.
• Retain failed jobs under a privacy-aware policy.
• Patch Redis, Node, BullMQ, and base images.
• Audit replay and administrative dashboard access.

Troubleshooting

Common Next.js Background Job Mistakes

Next.js Background Job Best Practices

• Queue only work that can safely finish after the response.
• Use a maintained queue or workflow product instead of inventing reliability primitives.
• Run consumers separately from Next.js web processes.
• Validate, authorize, minimize, and version every payload.
• Make every externally visible side effect idempotent.
• Retry only transient failures with bounded backoff and jitter.
• Control concurrency from measured downstream capacity.
• Use object storage for files and PostgreSQL for authoritative task state.
• Use an outbox where a database write must reliably cause a job.
• Keep Redis private with intentional persistence and eviction.
• Design graceful shutdown and deploy compatibility.
• Monitor age, depth, duration, failures, stalls, capacity, and business outcomes.

Related Next.js Production Tutorials

• Blog #10: Route Handlers for authenticated HTTP producer boundaries.
• Blog #13: Authentication for session and authorization design.
• Blog #14: Performance Optimization for measurement and resource budgets.
• Blog #16: Environment Variables & Security for server-only credentials.
• Blog #18: Forms & Validation for strict untrusted-input validation.
• Blog #19: PostgreSQL + Drizzle for task state and transactional outbox data.
• Blog #20: Docker + Traefik for the public HTTPS edge.
• Blog #21: Docker Compose for private service networks and persistent data services.
• Blog #22: Redis Caching & Rate Limiting for Redis security, memory, persistence, and shared provider limits.

Frequently Asked Questions

Current Official References

• Next.js self-hosting guide
• Next.js instrumentation and lifecycle guidance
• BullMQ connections
• BullMQ workers, concurrency, progress, and shutdown
• BullMQ retry and backoff
• BullMQ Job Schedulers
• Redis persistence
• Redis eviction policy

Next Steps

You now have a production model for asynchronous work: a trusted Next.js producer, a durable Redis-backed queue, separately deployed BullMQ workers, safe payloads, idempotent effects, bounded retries, controlled concurrency, and observable failures. Monitor that complete stack in Blog #24: Next.js 16 Monitoring & Observability.