Upgrades & Rollback

This page covers single-host Docker Compose installs: how to upgrade, what rollback really does to your data, and how to redeploy without losing tables. Helm installs use helm upgrade and helm rollback instead, see Kubernetes & Helm.

Install basics are in the Deployment Guide. Examples use /opt/poker-engine as the installation path.

Choose an upgrade path

	Admin Panel (host runner)	`upgrade.sh`
Starts from	Settings > License & Updates > Updates	An extracted update package on the server
Signed release manifest check	Yes, required	No
Rehearses migrations on a database copy	Yes	No
Boots a canary of the new version	Yes	No
Live session drain	Yes, when enabled	No
Rollback restores	Database, `.env.prod`, compose file, Nginx config and scripts	See `upgrade.sh` rollback limitations

Use the Admin Panel path whenever you can. It always moves to the latest release available on your license. Neither path can install a chosen older version.

docker-upgrade.sh is the script the host runner calls. Running it by hand stops with Signed release manifest verification is required, because only auto-update.sh performs that check and passes the result on. Use the Admin Panel, or run ./auto-update.sh as described below.

Before you upgrade

Read the release notes, using View Release Notes on the Overview tab.
Choose a quiet time. Players are disconnected during cutover, and the rollback data-loss window below grows with activity.
Check disk space. The updater refuses to continue with under 1024 MB free for the backup (POKER_ENGINE_MIN_BACKUP_FREE_MB) or under 2048 MB in /tmp for the download (POKER_ENGINE_MIN_UPDATE_FREE_MB). It also needs room to extract the package and load the image, and a temporary copy of your database is created on the PostgreSQL volume.
Make sure the backup service works. It must be defined in your compose file, and its status is in backups/postgres/backup-status.json. Rehearse a restore, which writes backups/postgres/restore-drill-status.json for the readiness checks:
docker compose --env-file .env.prod -f docker-compose.prod.yml run --rm backup restore --latest --rehearsal
Note your current version from curl -s http://127.0.0.1:3000/health. Rollback needs the running app to report it, and the old image must still be on the host, so don’t run docker image prune -a or docker system prune -a.
Copy backups off the server if you can. .env.prod, including ENCRYPTION_KEY, must be kept with them: encrypted data in a backup is unusable without that key.
Save your local changes to docker-compose.prod.yml and nginx/. Upgrades overwrite these. Certificates in nginx/ssl/ are kept, unless the package contains files with the same names.

Upgrade from the Admin Panel

Requirements on the host: the runner installed by install.sh or upgrade.sh, plus jq, Node.js, curl and cron. Button and warning details are in Automatic Updates.

Open Settings > License & Updates and go to the Updates tab.
Review Update Readiness Warnings, and resolve any you can.
Click Check Now (updates:manage). The cron entry runs the runner every 5 minutes, and the page shows Update in progress until it finishes. The check has to produce a verified signed manifest before you can continue.
Click Run Preflight (updates:manage). This takes a backup, downloads the package, loads the image, restores the backup to a temporary database, runs migrations on it and boots a canary on the new image, without touching the running site.
Click Apply Update (updates:apply), and confirm the versions, preflight and backup shown.
Watch Update Trace. On success the page shows Update completed.

A queued command expires if the runner doesn’t pick it up in time, 30 minutes by default. To run it right away instead of waiting for cron:

cd /opt/poker-engine
./auto-update.sh
tail -n 50 updater/last-run.log

What Apply Update does

Verifies the license and the signed manifest, and that the version and manifest hash match what you approved.
Runs the full preflight above.
Creates another pre-upgrade backup as backups/postgres/poker_engine_<timestamp>.sql.gz, and repeats the rehearsal.
Copies the new compose file, Nginx configuration, scripts and backup service files, rebuilds backup, and sets POKER_ENGINE_VERSION in .env.prod.
Drains live sessions if enabled.
Stops app and nginx, leaving PostgreSQL running. The site is down from here.
Runs prisma migrate deploy, starts app and nginx, and polls /health for up to 120 seconds.

If a migration fails or the app doesn’t become healthy, it rolls back automatically.

Automatic updates

Under Policy, updates are off by default (Automation Enabled: Disabled; Behavior: Notify only). With Auto-apply, the runner only installs a release when all of these are true:

The release’s channel matches Release Channel.
Its size is within Allowed Update Level. The default is Patch only.
The current UTC time is inside the Maintenance Window, if enabled.
The manifest doesn’t mark its migration as destructive or as needing manual approval. Such a release is skipped until you apply it yourself.

The maintenance window only limits automatic installs. Apply Update runs whenever you click it, and doesn’t enable maintenance mode for you.

Where records are kept

Path	Contents
`updater/runs/<run-id>/`	`run.json`, `preflight.log`, `apply.log`, stage outcome files, `canary.log`
`updater/state.json`	Last outcome and error, versions, preflight result, backup details
`updater/command-audit.log`	Every command’s lifecycle
`updater/runner.log`, `updater/last-run.log`	Runner output

Upgrade with `upgrade.sh`

Use this when the host runner isn’t an option. You need the update package for the new version, which contains image.tar.gz. The main package doesn’t include the image.

tar xzf poker-engine-X.Y.Z-update.tar.gz
cd poker-engine-X.Y.Z-update
./upgrade.sh /opt/poker-engine

Add --yes to skip the prompt. For an install that has never had a license, add --license-key and --license-domain.

The script validates .env.prod, verifies the license, writes backups/pre-upgrade-<version>-<timestamp>.sql, loads the image, copies the compose file, Nginx configuration and scripts, updates POKER_ENGINE_VERSION, stops app and nginx, runs migrations, starts them again and polls /health for up to 120 seconds.

It does not verify a signed manifest, rehearse migrations, or drain sessions: a stop still goes through the normal shutdown described in Live session migration, with the 10 second Docker stop limit. It also doesn’t refresh docker/backup/, scripts/backup-db.sh, scripts/restore-db.sh or scripts/release/, so copy those from the package yourself and rebuild with docker compose --env-file .env.prod -f docker-compose.prod.yml up -d --build backup.

Stuck migrations

If earlier failures left a migration marked rolled back, upgrade.sh warns before migrating and names it. If you have confirmed the schema is already in the expected state, mark it as applied:

docker compose --env-file .env.prod -f docker-compose.prod.yml run --rm --entrypoint "" migrations \
  npx prisma migrate resolve --applied <migration_name>

How rollback really works

Migrations only go forward. prisma migrate deploy applies pending migrations and there are no down migrations. An older app can’t be assumed to work on a migrated database, so going back always means restoring a database backup taken before the upgrade.

That has consequences:

Warning: A rollback discards everything written after the backup was taken. That includes hands, deposits, withdrawals, registrations and admin changes. The site keeps running during backup, download, image load and preflight, which can take many minutes, so this window is real. Plan the upgrade for a quiet time, and reconcile payment activity with your providers after any rollback.

Rolling back after a successful upgrade is a manual restore and loses all activity since the upgrade. The longer you wait, the more you lose. If the new version is merely misbehaving, a fix-forward with support may cost less.
Live sessions don’t survive a rollback. The pre-upgrade backup was taken before the drain, so it has no snapshots of those tables. In-progress hands in it are voided on start.
.env.prod keeps any changes you made after the backup, except what the rollback restores below.

Automatic rollback

When docker-upgrade.sh fails after cutover has begun, it:

Stops app and nginx.
Drops and recreates the database, then restores the pre-upgrade backup. It uses the backup container’s restore-db.sh if that container is running, or a direct psql restore otherwise.
Restores .env.prod, docker-compose.prod.yml, nginx/ and the deployment scripts, and sets POKER_ENGINE_VERSION back.
Starts app and nginx on the previous image, which is still on the host.
Polls /health for up to 120 seconds, and reports Upgrade failed. Rolled back to v<version>.

A failure before cutover, including a failed preflight, doesn’t touch the running site, so no rollback is needed. In the Admin Panel you see Update failed or rolled back and a Rollback stage in Update Trace.

Two things to check afterwards. The backup service files in docker/backup/ and scripts/ stay at the new version, as they are not part of the rollback. And if the version couldn’t be detected when the upgrade began, because the app wasn’t running, rollback has no version to return to: set POKER_ENGINE_VERSION yourself.

`upgrade.sh` rollback limitations

Treat the automatic rollback in upgrade.sh as incomplete. After it runs, always verify the result, even if it reports success:

It restores its plain SQL backup over the existing database without dropping it first and without stopping on errors, so the database may not actually return to its earlier state.
It restores docker-compose.prod.yml but leaves the new POKER_ENGINE_VERSION in .env.prod, so it can start the new image.

Check curl -s http://127.0.0.1:3000/health for the version, and if the rollback was needed, follow the manual steps below with the backups/pre-upgrade-<version>-<timestamp>.sql file.

Manual rollback

Use this when automatic rollback failed, or to go back after a completed upgrade.

Warning: This destroys the current database contents. Step 2 keeps a copy, so don’t skip it. You need the pre-upgrade backup and the previous version number. The old image must still be on the host, otherwise run docker load -i image.tar.gz from the previous version’s update package.

Stop the app and Nginx:
cd /opt/poker-engine docker compose --env-file .env.prod -f docker-compose.prod.yml stop app nginx
Keep a copy of the current database:
docker compose --env-file .env.prod -f docker-compose.prod.yml exec -T postgres \ pg_dump -U poker poker_engine | gzip > before-rollback-$(date +%Y%m%d-%H%M%S).sql.gz
Restore the pre-upgrade backup.
For a backup in backups/postgres/ (Admin Panel upgrades), list the files and restore with a reset. You are asked to confirm:
docker compose --env-file .env.prod -f docker-compose.prod.yml run --rm backup restore --list docker compose --env-file .env.prod -f docker-compose.prod.yml run --rm backup restore poker_engine_YYYYMMDD_HHMMSS.sql.gz --reset-database
For a plain backups/pre-upgrade-*.sql file from upgrade.sh, recreate the database and load it:
docker compose --env-file .env.prod -f docker-compose.prod.yml exec -T postgres \ psql -U poker -d postgres -v ON_ERROR_STOP=1 \ -c 'DROP DATABASE IF EXISTS poker_engine WITH (FORCE);' \ -c 'CREATE DATABASE poker_engine;' docker compose --env-file .env.prod -f docker-compose.prod.yml exec -T postgres \ psql -U poker -d poker_engine -v ON_ERROR_STOP=1 < backups/pre-upgrade-<version>-<timestamp>.sql
Use your own POSTGRES_USER and POSTGRES_DB values if you changed them.
Put the old configuration back. upgrade.sh and docker-upgrade.sh both leave docker-compose.prod.yml.bak:
cp docker-compose.prod.yml.bak docker-compose.prod.yml
Set the old version in .env.prod:
POKER_ENGINE_VERSION=X.Y.Z
Start, and verify the version:
docker compose --env-file .env.prod -f docker-compose.prod.yml up -d app nginx curl -s http://127.0.0.1:3000/health
Don’t run the migrations service here: the old version’s schema is already in place.

Then collect a support bundle and contact support before trying again.

Planned redeploys that keep tables alive

Live session migration carries active tables across a planned restart on a single app instance. It is off by default.

LIVE_MIGRATION_ENABLED=true
LIVE_MIGRATION_DRAIN_WINDOW_MS=90000
LIVE_MIGRATION_GRACE_MS=10000
LIVE_MIGRATION_DRAIN_TIMEOUT=120

Variable	Default	Read by	Meaning
`LIVE_MIGRATION_ENABLED`	`false`	App	Turns the feature on
`LIVE_MIGRATION_DRAIN_WINDOW_MS`	`90000`	App	How long the drain runs before snapshots are written
`LIVE_MIGRATION_GRACE_MS`	`10000`	App	Extra action timer time after a table is restored
`LIVE_MIGRATION_DRAIN_TIMEOUT`	`120`	`docker-upgrade.sh`	Seconds the updater waits for the app to exit. Keep it above the drain window

Recreate the app after changing these. /health then shows "liveMigrationEnabled": true.

What happens during a drain

/health/ready starts returning 503, /health reports "draining": true, and connected clients receive a server_migrating event.
New tables can’t be created and no new hands start. Players trying to take a seat are told: Server is updating. Your current hand is safe; new seats are temporarily paused.
Connected seated players get a disconnected-session record, so they can return to their seats.
At the deadline, active tables are paused and each table’s state is written to table_snapshots as pending. With a signal-triggered drain the process then exits.
The next start restores pending snapshots before it accepts traffic, and adds the grace time to the action timer.

The drain always runs for the full window, even if all hands finish early. Plan on a pause of at least LIVE_MIGRATION_DRAIN_WINDOW_MS, plus the time for migrations and start-up.

With upgrades

The Admin Panel path handles this for you. If the running app reports liveMigrationEnabled, the updater sends SIGUSR2 to the app and waits up to LIVE_MIGRATION_DRAIN_TIMEOUT seconds, then continues with the stop, migrations and start. If the wait times out, it continues with a normal stop, which writes snapshots as part of shutdown.

Set LIVE_MIGRATION_DRAIN_TIMEOUT in .env.prod, where the host runner reads it. upgrade.sh doesn’t run a drain.

Restart without an upgrade

Confirm the feature is on first. If it’s off, SIGUSR2 makes the app exit immediately with an error, and hands in progress are voided on start.

curl -s http://127.0.0.1:3000/health        # expect "liveMigrationEnabled":true

docker kill --signal=SIGUSR2 poker-engine-app

# Watch until the container stops, about one drain window
docker ps --filter name=poker-engine-app

docker compose --env-file .env.prod -f docker-compose.prod.yml up -d app
curl -s http://127.0.0.1:3000/health

Admin API

These need a super admin. Use your public URL with an admin access token, as shown in the Operations Runbook.

Request	Effect
`GET /api/v1/admin/system/drain`	Current state, including `snapshotsWritten`, `snapshotsRestored` and `orphanedHandsVoided` since the process started
`POST /api/v1/admin/system/drain`	Starts a drain. Optional JSON body: `windowMs` (0 to 300000) and `resumeExpectedMs` (0 to 3600000)
`POST /api/v1/admin/system/drain/finalize`	Writes snapshots now
`DELETE /api/v1/admin/system/drain`	Cancels a drain that hasn’t been finalized

A drain started this way doesn’t end the process. After finalizing, tables stay paused until a restart restores them, so restart the app promptly.

When things go wrong

Crash or kill without snapshots. On every start, whether or not the feature is on, the app voids in-progress games that have no pending snapshot. It sets voided with void_reason of server_failure, resets the players’ bets and records a hand_voided audit event.
A snapshot fails to restore. Its status becomes restore_failed and the data is kept for review. Check for this in the logs (Failed to restore live table snapshot), and contact support if you find any:
docker compose --env-file .env.prod -f docker-compose.prod.yml exec postgres \ psql -U poker -d poker_engine -c \ "SELECT id, table_id, status, reason, created_at FROM table_snapshots WHERE status = 'restore_failed';"
Checking a restored hand. After a restart with live migration, spot check tables with hands in progress in Game History and the audit trail, particularly whether a restored hand is marked voided.

Snapshots include private cards and deck order. Don’t select or share the payload column, and treat database backups accordingly.

Maintenance windows

Maintenance mode is a setting in Settings > Features > Maintenance Mode, or:

curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" https://your-domain.com/api/v1/admin/general/maintenance/enable
curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" https://your-domain.com/api/v1/admin/general/maintenance/disable

While it is on, non-admin API requests are blocked, and new non-admin WebSocket connections receive a MAINTENANCE error and are closed. Health and metrics routes keep working, and /health reports "maintenance": true. It takes up to 10 seconds to apply to the API. It doesn’t end sockets that are already connected or hands already running.

Suggested order for a window with downtime:

Tell players in advance. See Messages.
Enable maintenance mode shortly before the window, and let running hands finish. This also shrinks the rollback data-loss window, because almost nothing is being written.
Silence expected alerts. See Silencing.
Take and verify a backup, then upgrade.
Check /health for the new version, sign in as an admin and test a table.
Disable maintenance mode and watch logs and dashboards.

Don’t combine maintenance mode with live migration. Live migration relies on players reconnecting right after the restart, and maintenance mode refuses those connections. For a window that keeps tables alive, leave maintenance mode off and let the drain do the work.

After any upgrade

/health shows the expected version, with "status": "ok".
docker compose --env-file .env.prod -f docker-compose.prod.yml ps shows everything healthy, including backup.
Re-apply your local changes to docker-compose.prod.yml and nginx/.
License & Updates shows Update completed with no new warnings.
A real player flow works: log in, join a table, play a hand.
Remove old packages and backups you no longer need, once you are confident in the new version. Keep the previous image until then.