Runbook & Troubleshooting

Use this page during an incident on a single-host Docker Compose install. Each section gives the symptoms, how to diagnose them and how to fix them. Run commands from your installation directory, on the server.

Start-up failures, certificates and backup basics are in the Deployment Guide. Alert definitions are in Monitoring, Metrics and Alerting.

The examples use the default PostgreSQL user poker and database poker_engine. If you set POSTGRES_USER or POSTGRES_DB in .env.prod, use your values.

First five minutes

# Container state and health
docker compose --env-file .env.prod -f docker-compose.prod.yml ps

# Is the app up, and what does it report?
curl -s http://127.0.0.1:3000/health
curl -s http://127.0.0.1:3000/health/ready

# Recent app errors
docker compose --env-file .env.prod -f docker-compose.prod.yml logs --tail 200 app

# CPU, memory, disk
docker stats --no-stream
df -h

Result	Go to
/health returns 503 with "status": "unhealthy"	Database connection exhaustion
"status": "degraded"	Memory pressure
/health is fine, the site is slow	High load
/health is fine, tables won’t connect	WebSocket issues
license.status is not valid	License & Updates

In the Admin Panel, Settings > Logs has the Errors, Email Logs and Dead Letter Queue tabs. See Logs Hub.

Admin API calls in this page

Admin requests go through your public URL with an admin access token, the accessToken returned at admin login. A request authenticated this way doesn’t need a CSRF token.

curl -s -H "Authorization: Bearer $ADMIN_TOKEN" https://your-domain.com/health/circuits

Plain HTTP requests to 127.0.0.1:3000 are refused with 400 INVALID_HOST_HEADER on anything other than /health, /health/live and /health/ready. See Calling admin-only endpoints from the server.

Before you restart the app

Live table state is held in the app’s memory, so a restart affects players. Treat it as a deliberate step.

• With live migration off (the default), hands in progress are lost. On the next start the app marks them as voided with the reason server_failure, resets the players’ bets for those hands and records a hand_voided audit event.
• With live migration on, a normal stop writes table snapshots that the next start restores. Docker allows a container 10 seconds to stop by default. For a planned restart, drain first as described in Planned redeploys.

docker compose restart app does not re-read .env.prod. After changing the file, use:

docker compose --env-file .env.prod -f docker-compose.prod.yml up -d app

High load

Symptoms

• API responses slow down, and HighLatency or EventLoopLagHigh fires.
• Player actions time out, and lobby updates lag.
• Some players get HTTP 429 or 503 from Nginx.

Diagnosis

docker stats --no-stream

curl -s -H "Authorization: Bearer $ADMIN_TOKEN" -H "X-Forwarded-Proto: https" \
  http://127.0.0.1:3000/metrics/summary

# Slowest recent requests. threshold is in ms, limit is at most 100
curl -s -H "Authorization: Bearer $ADMIN_TOKEN" -H "X-Forwarded-Proto: https" \
  "http://127.0.0.1:3000/metrics/slow?threshold=1000&limit=20"

curl -s -H "Authorization: Bearer $ADMIN_TOKEN" -H "X-Forwarded-Proto: https" \
  http://127.0.0.1:3000/metrics/routes

Decide which case you have:

Finding	Likely cause
app near its 2 CPU limit, event loop lag high	The app is saturated. Node.js handles game logic on one main thread
One route dominates /metrics/slow	That route, or the query behind it
All routes slow, postgres near its 1 CPU limit	The database. See Database connection exhaustion
postgres and app are fine, players get 429 or 503	Nginx limits, described below

Remediation

1. Reduce new activity. In Settings > Features, turn off features you can spare, such as private games. The API equivalent is:
   curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \ https://your-domain.com/api/v1/admin/general/features/privateGames/disable
2. Reduce bot activity. Disable bots on the Bots page, or per bot with POST /api/v1/admin/bots/:botId/disable. There is no single "pause all bots" endpoint.
3. Pause or remove tables you don’t need in Game Management > Tables. See Tables.
4. Check Nginx limits if real users are being refused. The bundled configuration allows 10 requests per second per IP on /api/ with a burst of 20, 1 request per second with a burst of 5 on login, and 20 concurrent connections per IP. Players behind one shared address can hit the connection limit. The settings are in nginx/nginx.conf and nginx/conf.d/default.conf. Upgrades overwrite these files, so re-apply your changes afterwards.
5. Scale up. Raise the app limits under deploy.resources.limits in docker-compose.prod.yml and recreate it. Upgrades replace this file too. Compose installs run one app instance, so see Scaling and Multi-Node before planning anything larger.
6. Find query problems. Set QUERY_MONITOR=true in .env.prod and recreate the app to log a warning when a request runs more than QUERY_MONITOR_THRESHOLD queries (default 10). Turn it off again afterwards.

Database connection exhaustion

Symptoms

• /health returns 503, and /health/ready reports "database": false.
• API requests hang and then fail, and logs show connection pool timeouts.
• pe_db_errors_total rises with code p2024, connection_unreachable, connection_timeout or connection_closed.
• PostgresDown or DatabaseCircuitBreakerOpen fires.

Diagnosis

docker compose --env-file .env.prod -f docker-compose.prod.yml ps postgres
docker compose --env-file .env.prod -f docker-compose.prod.yml logs --tail 100 postgres
docker compose --env-file .env.prod -f docker-compose.prod.yml exec postgres pg_isready
df -h

Connections by state:

docker compose --env-file .env.prod -f docker-compose.prod.yml exec postgres \
  psql -U poker -d poker_engine -c \
  "SELECT state, count(*) FROM pg_stat_activity WHERE datname = 'poker_engine' GROUP BY state;"

Limit in force:

docker compose --env-file .env.prod -f docker-compose.prod.yml exec postgres \
  psql -U poker -d poker_engine -c "SHOW max_connections;"

Queries running longer than 30 seconds:

docker compose --env-file .env.prod -f docker-compose.prod.yml exec postgres \
  psql -U poker -d poker_engine -c \
  "SELECT pid, now() - query_start AS duration, state, left(query, 100) AS query FROM pg_stat_activity WHERE datname = 'poker_engine' AND state <> 'idle' AND now() - query_start > interval '30 seconds' ORDER BY duration DESC;"

Waiting for locks:

docker compose --env-file .env.prod -f docker-compose.prod.yml exec postgres \
  psql -U poker -d poker_engine -c \
  "SELECT pid, wait_event_type, wait_event, left(query, 100) AS query FROM pg_stat_activity WHERE datname = 'poker_engine' AND wait_event_type = 'Lock';"

Remediation

1. If the disk is full, free space first. PostgreSQL can’t write without it. Check backups/ and support-bundles/, then move old files off the server.
2. If one query is blocking everything, end that query.
   Warning: This rolls back the query’s transaction. Identify the pid carefully, and don’t use it on a query you can’t explain. Don’t use a blanket statement that ends every long-running or idle connection, because it also cuts the app’s own working connections.
   docker compose --env-file .env.prod -f docker-compose.prod.yml exec postgres \ psql -U poker -d poker_engine -c "SELECT pg_cancel_backend(12345);"
   Try pg_cancel_backend first, with the real pid. Use pg_terminate_backend only if the query doesn’t stop.
3. If the pool is too small for your load, adjust it. The app takes DATABASE_POOL_SIZE (production default 50) and DATABASE_POOL_TIMEOUT in seconds (default 10) from .env.prod, then recreate the app. Keep the pool under max_connections, leaving room for the backup service, the upgrade scripts and postgres-exporter. docker-compose.prod.yml doesn’t set max_connections, so PostgreSQL’s default applies unless you add it. Raising it also raises PostgreSQL’s memory use, and the postgres service is limited to 1 GB by default.
4. Reset a stale pool by recreating the app, after reading Before you restart the app.
5. Don’t reset the game-db-hot-path circuit breaker until /health/ready is healthy again. Breakers retry on their own, so see Payment failures for how they behave.

Memory pressure

Symptoms

• /health reports "status": "degraded" and the app is slow.
• docker stats shows app near its 2 GB limit.
• The app restarts by itself. restart: always brings it back, and hands in progress are voided.

Diagnosis

docker stats --no-stream

# Was the container killed for memory?
docker inspect poker-engine-app --format '{{.State.OOMKilled}} {{.State.ExitCode}} {{.RestartCount}}'

# Heap, RSS and history
curl -s -H "Authorization: Bearer $ADMIN_TOKEN" https://your-domain.com/health
curl -s -H "Authorization: Bearer $ADMIN_TOKEN" -H "X-Forwarded-Proto: https" \
  http://127.0.0.1:3000/metrics/memory

# Is the host itself short of memory?
free -h

In the /health admin response, data.checks.memory.details has heapUsedMB, heapTotalMB, rssMB and heapUsagePercent. The status turns critical at 95 percent heap usage combined with at least 1024 MB of RSS. A steadily rising line in /metrics/memory during quiet periods suggests a leak, while one that follows player numbers suggests a capacity problem.

Remediation

1. Reduce what is in memory. Remove unused tables in Game Management > Tables.
2. Give the app more memory. Increase memory under deploy.resources.limits for app in docker-compose.prod.yml, and recreate it. Upgrades replace this file, so re-apply the change afterwards.
3. Raise the Node.js heap limit along with the container limit. Add this to .env.prod, keeping it well below the container limit:
   NODE_OPTIONS=--max-old-space-size=3072
4. If restarting is the only way to recover, plan it using Before you restart the app, and collect a support bundle and the /metrics/memory output first, as a restart clears the history.

WebSocket issues

Symptoms

• Players can use the lobby, but tables don’t connect or keep reconnecting.
• WebSocketMassDisconnect fires.
• pe_ws_connections_active drops sharply.

Diagnosis

# Is the app healthy? If not, fix that first
curl -s http://127.0.0.1:3000/health

# Are Redis and Nginx up?
docker compose --env-file .env.prod -f docker-compose.prod.yml ps redis nginx

# Nginx configuration and logs
docker compose --env-file .env.prod -f docker-compose.prod.yml exec nginx nginx -t
tail -n 100 nginx/logs/error.log

# Socket.IO handshake. A 200 with a JSON "sid" means the path works
curl -s "https://your-domain.com/socket.io/?EIO=4&transport=polling"

# Certificate dates
echo | openssl s_client -connect your-domain.com:443 2>/dev/null | openssl x509 -noout -dates

If you run the monitoring stack, the Disconnect Reasons panel in Poker Engine WebSocket Realtime is the fastest way to narrow things down.

Disconnect reason	Likely cause
ping timeout	Clients stop answering within the 60 second ping timeout. Check network problems and event loop lag: an overloaded server can’t answer in time, see High load
transport close, transport error	Something between the client and the app cut the connection, such as a restart, or an external proxy or load balancer in front of Nginx
server namespace disconnect	The server closed the connection deliberately

Other checks:

Finding	Cause and fix
Players get a MAINTENANCE error and disconnect	Maintenance mode rejects non-admin WebSocket connections. Turn it off in Settings > Features
/health shows "draining": true	A live migration drain is running, and new seats are refused until it finishes
Many players behind one address can’t connect	The 20 connection per IP limit in Nginx, see High load
Origin or CORS errors in the browser console	CORS_ORIGINS in .env.prod doesn’t contain the exact origin players use
Certificate errors	Browsers refuse secure WebSockets with a self-signed or expired certificate. See SSL/TLS Setup
An external proxy in front of Nginx	It must pass the Upgrade and Connection headers and allow long idle times. Nginx itself allows 7 days on /socket.io/. Set TRUST_PROXY to the real number of hops
Redis is down	The Socket.IO adapter and caching are affected. Check REDIS_PASSWORD and docker compose ... logs redis

Remediation

Fix the cause you found. If a config file changed:

docker compose --env-file .env.prod -f docker-compose.prod.yml exec nginx nginx -t
docker compose --env-file .env.prod -f docker-compose.prod.yml restart nginx

Restarting Nginx disconnects every player briefly, and clients reconnect by themselves. It does not restart the game, so tables stay in memory.

Payment failures

Symptoms

• Deposits or withdrawals fail, or a balance doesn’t update after payment.
• WithdrawalErrorSpike or CircuitBreakerOpen fires for a payment-* circuit.
• pe_payment_provider_errors_total rises.

Diagnosis

1. Check the provider’s own status page. Look for provider errors in the logs:
   docker compose --env-file .env.prod -f docker-compose.prod.yml logs --tail 500 app | grep -iE "payment|webhook|circuit"
2. Check circuit breakers. Payment circuits are named payment-<provider>, for example payment-stripe, and open after 5 consecutive failures by default.
   curl -s -H "Authorization: Bearer $ADMIN_TOKEN" https://your-domain.com/health/circuits
3. Check webhook events. Failed signature checks often point to a wrong signing secret. A provider card in the Admin Panel warns No webhook signing secret configured when none is saved.
   curl -s -H "Authorization: Bearer $ADMIN_TOKEN" https://your-domain.com/api/v1/webhooks/events curl -s -H "Authorization: Bearer $ADMIN_TOKEN" https://your-domain.com/api/v1/webhooks/statistics
   Providers must be able to reach https://your-domain.com/api/v1/webhooks/payment/{paymentSystemId}.
4. Check the Dead Letter Queue in Settings > Logs > Dead Letter Queue, filtering by Pending or Failed. Permanently failed background operations are kept here.

Remediation

Payment settings are stored in the database and managed in the Admin Panel, not .env.prod. See Cashier & Payments.

1. If the provider is down, wait. An open circuit tries again by itself after its reset time: 60 seconds by default for payments. Consider telling players, and turning the method off for the duration.
2. If you changed credentials or the provider has recovered and the circuit is still open, you can reset it. Only do this when you are confident the provider is healthy, as an open circuit is protecting you from further failures:
   curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \ https://your-domain.com/api/v1/admin/circuits/payment-stripe/reset
   This requires the circuits:reset permission.
3. Replay missed webhooks after checking the event, with POST /api/v1/webhooks/events/:eventId/retry.
4. Retry dead letter items from the Dead Letter Queue tab, which needs settings:edit. Check the item first.
   Warning: Before retrying or adjusting anything involving money, compare the provider’s records with the transaction in the Admin Panel. Don’t credit a balance manually while a retry or webhook replay for the same payment is outstanding, or the player could be credited twice. Dismissing a dead letter item gives up on the operation.
5. If LedgerDrift fired as well, stop automated payouts and investigate before retrying anything further.

Email delivery

Symptoms

• Players don’t receive verification or password reset emails.
• CircuitBreakerOpen fires for email-service.
• Email Logs shows failures.

Diagnosis

1. Open Settings > Logs > Email Logs and read the error on a failed message.
2. In the SMTP settings, click Test Connection. It reports whether the connection and authentication succeeded. See SMTP Configuration. The API equivalent is:
   curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" https://your-domain.com/api/v1/admin/smtp/test
3. Check the circuit, and the logs:
   curl -s -H "Authorization: Bearer $ADMIN_TOKEN" https://your-domain.com/health/circuits docker compose --env-file .env.prod -f docker-compose.prod.yml logs --tail 500 app | grep -iE "email|smtp"
4. Check outbound access from the host, using your SMTP host and port. Many hosting providers block outbound mail ports by default.
   nc -zv smtp.example.com 587

Remediation

Finding	Fix
Authentication failure	Correct the Username / Password in SMTP settings. SMTP_* environment variables have no effect
Connection refused or timed out	Check the host, port and firewall, or ask your provider to open the port
Accepted, but mail never arrives	Check the provider’s logs, spam placement, and SPF and DKIM for your sending domain
Emails stop partway through the hour or day	You may have reached Max Per Hour or Max Per Day in the SMTP rate limits
Circuit open	It opens after 5 consecutive failures and tries again after 30 seconds. Fix the cause first. To reset it manually: POST /api/v1/admin/circuits/email-service/reset

After fixing the cause, retry failed messages from Email Logs, which needs settings:edit.

Support bundle

Collect this before restarting, and attach it when you contact info@onlinepokerscript.com:

./support-bundle.sh /opt/poker-engine

The script prints the path of the tarball, which goes in support-bundles/ under your installation unless you set POKER_ENGINE_SUPPORT_BUNDLE_DIR.

It contains:

• The updater’s configuration, state, command audit log and run records
• Recent runner, last-run and backup logs
• Backup and restore drill status
• docker-compose.prod.yml and the nginx/ directory
• .env.prod with passwords, secrets, keys, tokens and connection URLs replaced by <redacted>

It doesn’t contain the database, or application container logs. Add those logs yourself if relevant:

docker compose --env-file .env.prod -f docker-compose.prod.yml logs --tail 2000 app > app.log

Warning: The bundle copies the whole nginx/ directory, which includes your TLS private key at nginx/ssl/privkey.pem. Remove it before sending the bundle to anyone:

cd support-bundles
tar xzf poker-engine-support-<timestamp>.tar.gz
rm -rf poker-engine-support-<timestamp>/config/nginx/ssl
tar czf poker-engine-support-<timestamp>-clean.tar.gz poker-engine-support-<timestamp>
rm -rf poker-engine-support-<timestamp> poker-engine-support-<timestamp>.tar.gz

Look through the bundle before sharing it. .env.prod non-secret values such as your domain remain visible. When you contact support, include only the first 4 characters of your license key, the domain and the exact error message.

Restoring from a backup

This is a last resort for data damage. To undo a failed upgrade, see Upgrades, Rollback and Live Session Migration.

Warning: A restore replaces the whole database. Everything since the backup is lost, including hands, deposits, withdrawals and registrations. Reconcile payments with your providers afterwards.

1. List backups, and test the one you plan to use. A rehearsal restores into a disposable database, verifies it and drops it:
   docker compose --env-file .env.prod -f docker-compose.prod.yml run --rm backup restore --list docker compose --env-file .env.prod -f docker-compose.prod.yml run --rm backup restore poker_engine_YYYYMMDD_HHMMSS.sql.gz --rehearsal
2. Stop the app and Nginx:
   docker compose --env-file .env.prod -f docker-compose.prod.yml stop app nginx
3. Take a backup of the current state, in case you need something from it:
   docker compose --env-file .env.prod -f docker-compose.prod.yml run --rm backup backup
4. Restore. --reset-database ends connections, then drops and recreates the database. You are asked to confirm:
   docker compose --env-file .env.prod -f docker-compose.prod.yml run --rm backup restore poker_engine_YYYYMMDD_HHMMSS.sql.gz --reset-database
5. Apply any migrations and start again:
   docker compose --env-file .env.prod -f docker-compose.prod.yml run --rm migrations docker compose --env-file .env.prod -f docker-compose.prod.yml up -d app nginx curl -s http://127.0.0.1:3000/health

A backup from an older version needs step 5’s migrations, or the matching older image. See How rollback really works.

Related

• Monitoring, Metrics and Alerting
• Upgrades, Rollback and Live Session Migration
• Deployment Guide
• Logs Hub
• Cashier & Payments
• System Emails