Custom Background Workers

Frappe runs background jobs in three standard queues: short, default and long. On a private bench group, you can add your own queues. Each custom queue gets its own worker.

Use a custom queue when your app has slow jobs that must not block the standard queues. For example, a long import job in a custom queue does not delay emails and other jobs in the default queue.

The framework docs explain queues and workers in detail. See Background Jobs.

Add a custom queue

1. In the dashboard, open your bench group.
2. Open the Config tab.
3. Click Add Config.
4. In Config Name, select Custom Key.
5. In Key, type workers.
6. In Type, select JSON.
7. In Value, enter your queues. The example below adds one queue with the name myqueue.
8. Click Add Key.
9. Deploy the bench group.

{
  "myqueue": {
    "timeout": 3000
  }
}

The worker starts only after the deploy is complete. A change to the config alone does not start or stop a worker. Jobs that you send to the queue before the deploy stay in the queue until the worker starts.

Send jobs to the queue

In your app, give the queue name to frappe.enqueue:

frappe.enqueue("my_app.tasks.import_records", queue="myqueue")

Settings for each queue

Frappe Cloud starts one worker for each custom queue. The framework setting background_workers has no effect for a custom queue on Frappe Cloud.

Remove a custom queue

1. Remove the queue from the workers value, or delete the workers key.
2. Deploy the bench group.

Before you remove a queue, make sure that your app does not send jobs to it. If your app sends a job to a queue that is not in the config, frappe.enqueue fails with the error "Queue should be one of ...".

Logs

Each custom queue writes to its own log files in the logs folder of the bench: myqueue.log and myqueue.error.log. To read them, see Debugging.