Workers

Table of contents
  1. Workers
    1. The queue:work command
    2. Running the worker inside your application
    3. Queue priorities
    4. Queue workers and deployment
    5. Reacting to worker signals
    6. Job expiration and timeouts
    7. Pausing and resuming
    8. Maintenance mode
    9. Resource considerations
    10. Supervisor

The queue:work command

perago queue:work                        # default connection, its default queue
perago queue:work redis --queue=emails   # a connection and a queue
perago queue:work --name=mailer          # a name for logs and supervisors

From Go, the same flags are perago.WorkerOptions:

status, err := m.Work(ctx, perago.WorkerOptions{
	Connection: "redis", Queues: []string{"high", "default"},
	Tries: 3, Backoff: []time.Duration{10 * time.Second}, Timeout: time.Minute,
	Sleep: 3 * time.Second, MaxJobs: 1000, MaxTime: time.Hour, MemoryMB: 256,
	HandleSignals: true,
})

Workers are long-lived processes, and code changes need a restart (see deployment). queue:listen starts a fresh process for each job instead: slower, but it picks up new code on its own.

Flag Meaning
--queue=high,default Queues in priority order
--once Process one job, or none, and exit
--stop-when-empty Exit once every queue is empty
--max-jobs=1000 Exit after this many jobs
--max-time=3600 Exit after this many seconds
--sleep=3 Seconds to wait when the queues are empty
--rest=0 Seconds to rest after each job
--tries=1 Attempts for jobs without Tries() (0 = forever)
--backoff=10,30 Backoff for jobs without Backoff()
--timeout=60 Seconds a job may run (0 = no limit)
--memory=128 or --memory=60% Exit (status 12) past this memory
--force Work even while App.Maintenance reports maintenance mode

Durations take seconds or Go durations (90s, 2m).

Running the worker inside your application

A worker does not need a process of its own. m.Work blocks until the context is cancelled, so start it in a goroutine of your service: the application dispatches jobs and works them, with nothing but the database (or Redis) next to it.

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()

go func() {
	if _, err := m.Work(ctx, perago.WorkerOptions{Queues: []string{"high", "default"}}); err != nil {
		slog.Error("worker stopped", "err", err)
	}
}()

http.ListenAndServe(":8080", handler) // the handlers call m.Dispatch(...).Send(ctx)

Start several goroutines for more parallelism, each with its own m.Work call. When the context is cancelled, each worker finishes its current job and returns. With more than one process (several instances, or separate queue:work processes), give the manager a shared cache: see Choosing a backend.

Queue priorities

--queue=high,low takes every job from high before looking at low.

Queue workers and deployment

Workers keep your code in memory. After a deploy, run perago queue:restart: every worker exits after its current job, and your process manager starts new ones. The restart signal lives in the Manager’s Cache, so the cache must be shared by every worker, which means database or Redis.

Reacting to worker signals

With HandleSignals (always on for queue:work):

  • SIGTERM, SIGINT, SIGQUIT: finish the current job, then exit. A job that implements Interrupted(os.Signal) is told right away, so it can stop early.
  • SIGUSR2: pause. SIGCONT: continue.

Job expiration and timeouts

retry_after is the time a reserved job may run before another worker takes it again: RetryAfter for database and Redis, the TTR for Beanstalkd, the visibility timeout for SQS. Keep it longer than the longest job timeout, or a slow job runs twice.

The worker --timeout (or the job’s Timeout()) cancels the job’s context once it expires. The job counts as failed with JOB_TIMEOUT: it is retried with backoff, or failed for good when FailOnTimeout is set or its attempts are used up. Go cannot kill a goroutine. A job that ignores its context and outlives WorkerOptions.Grace (5s) makes the worker exit with status 1, so the process manager can replace it, much like Laravel’s worker killing itself on timeout. Give your HTTP clients and sockets timeouts of their own.

Pausing and resuming

perago queue:pause database:default     # workers finish their job, then take nothing from it
perago queue:pause --all                # every queue
perago queue:continue database:default  # or queue:resume
perago queue:resume --all               # lifts --all; individually paused queues stay paused

Workers check for restart and pause signals in the cache before each job. If you don’t need these features, skip the checks with m.WithoutInterruptionPolling(), or with WorkerOptions.SkipRestartPolling and SkipPausePolling.

Maintenance mode

Set console.App.Maintenance (or WorkerOptions.Maintenance) to a function that reports maintenance mode. Workers wait while it returns true, unless they run with --force.

Resource considerations

Release large resources at the end of each job: close files and image handles, and return pooled connections. --max-jobs and --memory let a leaky worker restart itself.

Supervisor

Run workers under a process manager that restarts them: Supervisor, systemd, Kubernetes or Nomad. A Supervisor program looks like this:

[program:perago-worker]
process_name=%(program_name)s_%(process_num)02d
command=/srv/app/bin/app queue:work redis --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=app
numprocs=8
redirect_stderr=true
stdout_logfile=/srv/app/storage/logs/worker.log
stopwaitsecs=3600

stopwaitsecs must be longer than your longest job, so Supervisor waits for it instead of killing it. Then:

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start "perago-worker:*"