This page is for operators of a single-host Docker Compose install. It covers what you can monitor with no extra setup, how to turn on Prometheus metrics, and how to run the optional monitoring stack with its alert rules, dashboards and tracing.
Install basics, .env.prod and the health endpoint overview are in the Deployment Guide. Run the commands below from your installation directory.
What is available out of the box
| Signal | Where | Access |
|---|---|---|
| Health | /health, /health/live, /health/ready | Public, through Nginx or on 127.0.0.1:3000 |
| Circuit breakers | /health/circuits | Admin session or admin access token |
| Performance JSON | /metrics, /metrics/summary, /metrics/memory, /metrics/slow, /metrics/routes, /metrics/timers | Admin only, not proxied by Nginx |
| Error, email and recovery history | Settings > Logs in the Admin Panel, see Logs Hub | settings:view permission |
| Update and backup readiness | Settings > License & Updates, see License & Updates | updates:view permission |
| Container logs | docker compose --env-file .env.prod -f docker-compose.prod.yml logs -f app | Host shell |
Reading /health
curl -s https://your-domain.com/health| Field or status | Meaning |
|---|---|
HTTP 200, "status": "ok" | Database and memory checks pass |
HTTP 200, "status": "degraded" | The database is fine but memory is critical |
HTTP 503, "status": "unhealthy" | The database check failed |
license.status | Current license state |
maintenance | Maintenance mode is on |
liveMigrationEnabled, draining, activeHands, drainDeadlineAt | Live session migration state, see Upgrades, Rollback and Live Session Migration |
Memory counts as critical only when heap usage is at least 95 percent and process RSS is at least 1024 MB. A request made as an admin also gets the database and memory detail under data.checks, plus migration and any license failure reason.
/health/ready returns 503 when the database is down, the license is not configured or the app is draining. /health/live returns 200 as long as the process can respond.
Calling admin-only endpoints from the server
Only /health, /health/live and /health/ready are exempt from the app’s HTTPS enforcement. In production, any other plain HTTP request is refused with 400 INVALID_HOST_HEADER unless it looks like it came through a proxy. To call the /metrics routes on 127.0.0.1:3000, add the header Nginx would normally set:
curl -s -H "Authorization: Bearer $ADMIN_TOKEN" \
-H "X-Forwarded-Proto: https" \
http://127.0.0.1:3000/metrics/summaryThis relies on TRUST_PROXY=1, the value in .env.prod.example. $ADMIN_TOKEN is the accessToken returned when an admin logs in. Access tokens are short-lived.
/health/circuits goes through Nginx, so use the public URL:
curl -s -H "Authorization: Bearer $ADMIN_TOKEN" https://your-domain.com/health/circuitsEnable Prometheus metrics
GET /metrics/prometheus stays dark until you set a token. Without one it returns 404. With one it returns 401 unless the request carries the exact bearer token. Default Node.js process metrics are only collected when the token is set.
- Generate a token and add it to
.env.prod. It must be at least 32 characters:openssl rand -hex 32METRICS_SCRAPE_TOKEN=your-generated-token - Recreate the app so it reads the new value. This restarts the app, so see Before you restart the app:
docker compose --env-file .env.prod -f docker-compose.prod.yml up -d app - Test the endpoint from the server:
curl -s -H "Authorization: Bearer $METRICS_SCRAPE_TOKEN" \ -H "X-Forwarded-Proto: https" \ http://127.0.0.1:3000/metrics/prometheus | head
Nginx does not proxy /metrics, so the endpoint is not reachable from the internet with the bundled configuration. To scrape from your own Prometheus, it must reach the app’s port and send the same two headers.
Run the monitoring stack
Getting the files
The stack is defined by docker-compose.observability.yml and the deploy/observability/ directory. They are not part of the Docker release packages. They are in the Enterprise source release. If you don’t have them, contact info@onlinepokerscript.com. Copy both into your installation directory, because the overlay uses paths relative to it.
The upgrade scripts only replace docker-compose.prod.yml, so your edits to these files survive upgrades, and the scripts never restart the monitoring containers.
What runs
| Service | Image | Host address | Purpose |
|---|---|---|---|
prometheus | prom/prometheus:v2.55.1 | 127.0.0.1:9090 | Scrapes metrics and evaluates alert rules every 15 seconds |
alertmanager | prom/alertmanager:v0.27.0 | 127.0.0.1:9093 | Routes alerts |
grafana | grafana/grafana:10.4.18 | 127.0.0.1:3001 | Dashboards |
otel-collector | otel/opentelemetry-collector-contrib:0.116.1 | 127.0.0.1:4317, 127.0.0.1:4318 | Receives traces |
jaeger | jaegertracing/all-in-one:1.62.0 | 127.0.0.1:16686 | Stores and shows traces |
postgres-exporter | prometheuscommunity/postgres-exporter:v0.16.0 | Internal | PostgreSQL metrics |
redis-exporter | oliver006/redis_exporter:v1.67.0 | Internal | Redis metrics |
Every published port is bound to loopback. Reach the interfaces over an SSH tunnel rather than opening them publicly:
ssh -L 3001:127.0.0.1:3001 -L 9090:127.0.0.1:9090 -L 9093:127.0.0.1:9093 -L 16686:127.0.0.1:16686 you@your-serverSetup
- Add these to
.env.prod, alongsideMETRICS_SCRAPE_TOKEN. IfGRAFANA_ADMIN_PASSWORDis missing, Grafana falls back to the passwordadmin.GRAFANA_ADMIN_USERis optional and defaults toadmin.GRAFANA_ADMIN_PASSWORD=your-generated-password - Write the scrape token where Prometheus expects it. The
secrets/directory is mounted read-only, and the file must be readable by the Prometheus container, whose image runs as usernobody(UID 65534):mkdir -p deploy/observability/prometheus/secrets grep '^METRICS_SCRAPE_TOKEN=' .env.prod | cut -d= -f2- | tr -d '\n' > deploy/observability/prometheus/secrets/metrics-token sudo chown 65534:65534 deploy/observability/prometheus/secrets/metrics-token sudo chmod 0400 deploy/observability/prometheus/secrets/metrics-token - Add the proxy header to the
poker-enginejob indeploy/observability/prometheus/prometheus.yml. This step is required in production. Prometheus scrapeshttp://app:3000directly, so without the header the app answers400 INVALID_HOST_HEADER, the target shows as down andBackendDownfires:- job_name: poker-engine metrics_path: /metrics/prometheus http_headers: X-Forwarded-Proto: values: ["https"] authorization: type: Bearer credentials_file: /etc/prometheus/secrets/metrics-token static_configs: - targets: - app:3000 - Change the OTLP exporter in
deploy/observability/otel-collector/otel-collector.yml. The shipped file uses theloggingexporter, which the collector removed in v0.111.0, so version 0.116.1 refuses to start with it. Replace it withdebugin both places:exporters: otlp/jaeger: endpoint: jaeger:4317 tls: insecure: true debug: verbosity: basic service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp/jaeger, debug] - Replace the placeholders in
deploy/observability/alertmanager/alertmanager.yml.example. Compose mounts this exact file as the live configuration, so edit it in place. See Alert routing. - Apply the two alert rule corrections in Known rule corrections.
- Start the production stack with the overlay:
docker compose --env-file .env.prod -f docker-compose.prod.yml -f docker-compose.observability.yml up -d - Open
http://127.0.0.1:9090/targetsthrough your tunnel. Thepoker-engine,postgres-exporter,redis-exporterandprometheusjobs should all be up.
Use both -f files and --env-file for every command that touches the overlay. To apply later changes to Prometheus configuration or rules, reload it:
curl -X POST http://127.0.0.1:9090/-/reloadDashboards
Grafana provisions six dashboards into the Poker Engine folder, using Prometheus as the default data source.
| Dashboard | Panels |
|---|---|
| Poker Engine Platform Overview | Backend Up, HTTP Request Rate, HTTP Error Rate, HTTP p95 Latency, Active Realtime And Tables, Circuit Breaker State, Node Event Loop Lag p95, Heap Used |
| Poker Engine HTTP API | Requests By Route, Status Codes, Latency p50/p95/p99, Active Requests, Slowest Routes p95 |
| Poker Engine WebSocket Realtime | Active Connections, Inbound Events, Outbound Events, Handler Latency p95, Disconnect Reasons, Broadcast Fanout p95 |
| Poker Engine Game Engine | Hands Started / Completed, Active Tables And Seats, Hand Duration p95, Pot Size p95/p99, Action Mix And Timeouts, Showdowns, Aborts, Invariants |
| Poker Engine Economy | Rake Collected, Deposits By Provider, Withdrawals By Provider, Balance Mutations, Ledger / Fraud / Chargebacks, Player Win Distribution |
| Poker Engine Infrastructure | Postgres Up And App Errors, Prisma Query Latency p95, Redis App And Exporter Health, Redis Command Latency p95, Database Pool And Migrations, Node Process |
What to watch
| Question | Metric | Look for |
|---|---|---|
| Are players getting errors? | pe_http_requests_total | Share of status values matching 5.. |
| Is the API slow? | pe_http_request_duration_seconds | p95 above 2 seconds |
| Is the process saturated? | nodejs_eventloop_lag_seconds | Sustained values above 0.2 |
| How much is connected? | pe_ws_connections_active, pe_tables_active, pe_seats_occupied | Your normal range, and sudden drops |
| Are connections dropping? | pe_ws_disconnects_total | Spikes, broken down by reason |
| Are hands flowing? | pe_hands_started_total, pe_hands_completed_total, pe_hands_aborted_total, pe_action_timeouts_total | Completions stopping while seats are occupied |
| Is the database healthy? | pe_db_errors_total, pe_db_query_duration_seconds, pg_up | Any connection_* code, or p2024 |
| Is Redis connected? | pe_redis_connected, pe_redis_reconnects_total | A value of 0, or reconnects climbing |
| Are dependencies failing? | pe_circuit_breaker_state | 0 is closed, 1 half-open, 2 open |
| Are payments working? | pe_deposits_total, pe_withdrawals_total, pe_payment_provider_errors_total | Failures by provider |
| Is money consistent? | pe_ledger_drift_detected_total, pe_gameplay_invariant_violations_total | Anything above 0 |
| Is memory growing? | nodejs_heap_size_used_bytes, process_resident_memory_bytes | Steady growth toward the 2 GB app limit |
pe_db_pool_connections reports the configured pool size, not live usage. To see real connection use, query PostgreSQL as shown in Database connection exhaustion.
Also keep an eye on things the metrics don’t cover: free disk space, certificate expiry, and backups/postgres/backup-status.json, which the backup service updates after each backup.
Alert rules
Rules live in deploy/observability/prometheus/rules/.
Outages (outages.yml)
| Alert | Fires when | Severity | Team | First response |
|---|---|---|---|---|
BackendDown | The app’s scrape target is down for 1 minute | critical | platform | Check ps and app logs. Confirm the scrape headers if this happens straight after setup |
PostgresDown | pg_up is 0 or the app sees connection-class errors, for 1 minute | critical | platform | Database connection exhaustion |
RedisDown | pe_redis_connected is 0 for 1 minute | critical | platform | Check the redis container and REDIS_PASSWORD |
HighErrorRate | 5xx responses above 5 percent for 5 minutes | critical | platform | Open Poker Engine HTTP API, find the route, then check logs |
HighLatency | HTTP p95 above 2 seconds for 10 minutes | warning | platform | High load |
CircuitBreakerOpen | Any circuit is open for 2 minutes | warning | platform | Check /health/circuits and whether the dependency is really down |
DatabaseCircuitBreakerOpen | A circuit with database or db in its name is open for 2 minutes | critical | platform | The game-db-hot-path circuit protects game persistence. Check database health first |
EventLoopLagHigh | Event loop lag above 200 ms for 5 minutes | warning | platform | High load |
WebSocketMassDisconnect | Disconnect rate above 3 times the one-hour rate for 5 minutes | warning | platform | WebSocket issues |
HandsStalled | Seats are occupied but no hands complete for 10 minutes | critical | game-integrity | Open Poker Engine Game Engine, find the table in logs, and pause it in Game Management > Tables if it is stuck |
Fraud and finance (fraud.yml)
| Alert | Fires when | Severity | Team | First response |
|---|---|---|---|---|
FraudAlertBurst | More than 5 fraud or AML alerts in 15 minutes | warning | finance | Review fraud alerts, AML reviews and recent deposits and withdrawals |
CriticalFraudAlert | Any critical fraud or AML alert | critical | finance | Apply your freeze policy, keep the audit evidence, review before releasing funds |
ChargebackSpike | More than 3 chargebacks in an hour | warning | finance | Review provider webhooks and the affected payment systems |
WithdrawalErrorSpike | Over 20 percent of withdrawals have status failed or error for 15 minutes | warning | finance | Payment failures |
LedgerDrift | Ledger reconciliation found drift | critical | finance | Stop automated payouts, export the affected entries, keep the alert active until explained |
Game integrity (anomalous-wins.yml)
| Alert | Fires when | Severity | Team | First response |
|---|---|---|---|---|
AnomalousWinAmount | A single-hand win exceeds 1,000,000 chips | warning | game-integrity | Find the hand in Game History and review the account |
GameplayInvariantViolation | A runtime chip, pot or payout invariant failed | critical | game-integrity | Use tableId and handNumber from the logs, and preserve the hand history before restarting |
RakeAnomalyLow | Rake per hand drops over 50 percent against the 24-hour baseline for 30 minutes | warning | finance | Check rake settings, table templates and private table overrides |
WinRateAnomaly | Pot-size p99 above 5 times the 24-hour baseline for 15 minutes | warning | game-integrity | Rule out legitimate high-stakes play, then review accounts and tables |
GameplayInvariantViolation depends on RUNTIME_INVARIANTS_ENABLED, which is on by default.
The alerts’ runbook annotations link to a private repository you can’t open. Use this page and the Operations Runbook instead.
Known rule corrections
Two shipped rules compare series with different labels, so as written they never fire. Replace their expr lines.
In rules/outages.yml, HandsStalled:
expr: sum(rate(pe_hands_completed_total[10m])) == 0 and sum(pe_seats_occupied) > 0In rules/anomalous-wins.yml, AnomalousWinAmount:
expr: sum(increase(pe_player_win_amount_chips_bucket{le="+Inf"}[5m])) - sum(increase(pe_player_win_amount_chips_bucket{le="1000000"}[5m])) > 0The 1,000,000 chip threshold is a fixed value. Adjust it to suit your stakes, using one of the metric’s bucket boundaries: 1000000, 5000000 or 10000000.
Alert routing
The example Alertmanager configuration routes by labels:
severity="critical"goes to thepagerwebhook, and then continues to the team routes.team="finance",team="game-integrity"andteam="platform"go to their own Slack receivers.- Anything else goes to
default-email.
Replace every placeholder: the example.com addresses and SMTP host, CHANGE_ME, the pager.example.com URL and the hooks.slack.com/services/CHANGE/ME URLs. A single operator can point every receiver at the same inbox or webhook. Leave the team labels in place so you can split routing later.
Alertmanager has no connection to the SMTP settings in the Admin Panel. It needs its own mail or webhook details, and this file then holds a credential, so restrict its permissions.
Silencing
Silence an alert in Alertmanager (http://127.0.0.1:9093) only for a known maintenance window, a provider incident you are already tracking, or a planned load test. Give every silence an owner and an expiry. An upgrade stops the app, so BackendDown fires if the stop lasts more than a minute: silence it for the window. Don’t silence LedgerDrift or GameplayInvariantViolation unless the person responsible for finance or game integrity accepts the risk.
Tracing
Tracing is off by default. The Node.js SDK starts only when OTEL_ENABLED=true, and sends traces only when OTEL_EXPORTER_OTLP_ENDPOINT is set.
OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_TRACES_SAMPLER_ARG=0.1| Variable | Default | Notes |
|---|---|---|
OTEL_ENABLED | false | Must be the exact value true |
OTEL_EXPORTER_OTLP_ENDPOINT | Not set | The base URL of the collector. Don’t include /v1/traces: the exporter appends that path itself, so the commented value in .env.prod.example results in a wrong URL |
OTEL_TRACES_SAMPLER_ARG | 0.1 | Share of traces kept, with the parentbased_traceidratio sampler |
Recreate the app after changing these. Then open Jaeger at http://127.0.0.1:16686 and choose the service poker-engine. The collector forwards traces to Jaeger. Jaeger’s all-in-one image is not given a persistent volume here, so treat traces as short-lived.
When tracing is active, application log lines include trace_id and span_id, so you can move from a log line to its trace. /health and the metrics routes behave the same with tracing on or off.
Error tracking
For exception reporting, set SENTRY_ENABLED and SENTRY_DSN. See Sentry Integration.