Out-of-Memory Issues
If an agent job fails due to the error
Traceback (most recent call last):
File "/home/frappe/agent/repo/agent/base.py", line 50, in execute
output, returncode = self.run_subprocess(
File "/home/frappe/agent/repo/agent/base.py", line 103, in run_subprocess
raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command 'docker exec -w /home/frappe/frappe-bench bench-xxxxx-000xxxx-fxx-reg bench --site site.frappe.cloud migrate --skip-search-index --skip-failing' returned non-zero exit status 137.
it means that your server ran out of memory and should ideally be upgraded to ease the load on the server. To upgrade your dedicated server refer this guide
Site update or migrate failed with out of memory (exit code 143 or 137): what to do
A site update, version upgrade or migrate on Frappe Cloud can stop with returned non-zero exit status 137 or returned non-zero exit status 143. Both codes mean that the server ran out of memory and the system stopped the migrate process.
The error
You see the error in the failed job. To find it, open the site, go to the Updates tab, and click View Job on the failed update. On a dedicated server, you also get a notification with the title Server out of memory error.
The error looks like one of these:
subprocess.CalledProcessError: Command 'docker exec -w /home/frappe/frappe-bench bench-xxxx-000xxx-fxx-reg bench --site example.frappe.cloud migrate --skip-search-index --skip-failing' returned non-zero exit status 137.
subprocess.CalledProcessError: Command 'docker exec -w /home/frappe/frappe-bench bench-xxxx-000xxx-fxx-reg bench --site example.frappe.cloud migrate' returned non-zero exit status 143.
The job output can also show only this line:
b'Terminated\n'
What exit codes 137 and 143 mean
The bench migrate command runs all database patches and schema changes of your apps. Large sites and major version changes use a lot of memory during migrate. All sites and background jobs on the same server share the same memory.
When the free memory on the server is too low, a memory guard stops the process that uses the most memory. This is often the migrate process.
- Exit code 143 means that the process got a stop signal (SIGTERM). The memory guard on Frappe Cloud servers sends this signal first.
- Exit code 137 means that the process was killed at once (SIGKILL). If the memory is still too low after the first signal, the memory guard sends this signal. The operating system can also send it.
The cause is the same for both codes. The migrate did not have enough memory to finish. Your data and your app code did not cause the stop.
What happens to your site after the failure
Before an update starts, Frappe Cloud takes a backup of the site. If the update fails, Frappe Cloud starts a recovery by itself. The recovery moves the site back to its old bench. Then it restores the database tables that the update changed. Last, it takes the site out of maintenance mode.
On the Updates tab, the update status shows one of these values:
| Status | What it means | What you do |
|---|---|---|
| Recovering | The recovery is in progress. | Wait. Do not start another update or migrate. |
| Recovered | The site is back on the old version and works again. | Reduce the memory load, then update again. See the sections below. |
| Fatal | The recovery did not finish, or the update skipped the backup. The site status is Broken. | Raise a support ticket. See Site is stuck after the failure. |
What you can do on a shared plan
A site on a shared plan runs on a server that Frappe Cloud manages. You cannot change the memory of that server.
- Make sure that the update status is Recovered.
- Uninstall the apps that the site does not use. See Reduce the memory load.
- Start the update again at a time with low traffic. In the update dialog, set Schedule Time in IST to a quiet hour.
- If the update fails again with the same exit code, raise a support ticket. Attach the site name and a screenshot of the failed job.
If the site is large or has many custom apps, a shared server can be too small for it. In this case, move the site to a private bench on a dedicated server. See Move site to a private bench.
What you can do on a dedicated server
On a dedicated server, the memory belongs to your sites only. You control how much memory the server has.
- Make sure that the update status is Recovered.
- Open the server in the dashboard and look at the memory graph on the Analytics tab. If memory use is near the limit during normal work, the server is too small for the update.
- Reduce the memory load. See Reduce the memory load.
- If the memory is still too low, upgrade the server plan. See Change the server plan.
- Start the update again.
After a successful update, you can change the server back to the smaller plan.
Reduce the memory load
These actions lower the memory that the migrate needs, or free memory for it.
- Uninstall apps that the site does not use. Each installed app adds patches and schema changes to the migrate. Go to the Apps tab of the site, open the menu of the app, and click Uninstall. Take a backup first, because uninstalling an app deletes its data.
- Retry at a time with low traffic. Users, reports and background jobs use memory too. In the update dialog, set Schedule Time in IST to a quiet hour.
- Update sites one at a time. If many sites on the same server update together, their migrations run at the same time and share the memory. Schedule the updates for different times.
- Split large updates on a private bench. If many apps have updates, deploy only some of them in one deploy. Update the site, then deploy the next apps.
- Do not skip the backup. If you select Skip taking backup for this update and the update fails, Frappe Cloud cannot roll back the site.
Major version upgrades (for example, v15 to v16)
A major version upgrade runs many more patches than a normal update. Some patches change large tables. A site that updates without problems on minor versions can run out of memory on a major upgrade.
- Upgrade one major version at a time. Frappe Cloud does not support an upgrade from v14 to v16 in one step.
- Test the upgrade on a copy of the site first. Restore a recent backup to a new site on a bench with the new version, and run the upgrade there.
- Before you upgrade a large site, add memory to a dedicated server for the duration of the upgrade.
- Start the upgrade from the Actions tab of the site, with Version upgrade. Do not select Skip backups.
Site is stuck after the failure
Sometimes the site stays in maintenance mode, or the site status stays Broken, Updating or Recovering.
- If the update status is Recovering, wait for the recovery to finish. A large database can take more than one hour.
- If the update status is Fatal, do not run another update or migrate. Frappe Cloud blocks new updates on this site and shows this message:
Site has encountered a fatal error during last update. Please open a ticket on our support portal with the error details to resolve the issue.
- Raise a ticket on the support portal. Attach the site name, the time of the failure, and a screenshot of the failed job.
The Activate action on the Actions tab does not fix a site with a fatal update. The support team must first restore the site, and only then can you update it again.
Raise a ticket also in these cases:
- The site status stays Recovering or Updating for more than four hours.
- The update failed with Skip taking backup for this update selected. In this case, no backup exists to roll back to.
How to prevent it
- Keep only the apps that the site uses.
- Schedule updates for a time with low traffic.
- On a dedicated server, look at the memory graph before a major version upgrade. If memory use is near the limit, add memory.
- Test major version upgrades on a copy of the site first.