Lunch rush timeouts
Tablekeep is a restaurant reservation platform used by 9,000 restaurants. A Rails monolith runs on 12 Puma app servers behind an nginx load balancer and Cloudflare. All app servers reach Postgres 14 through PgBouncer in transaction mode (pool of 80 server connections to the primary); a streaming read replica serves the public restaurant pages and menus. Redis holds sessions and the page cache, Sidekiq workers on two worker servers send SMS reminders through Twilio and confirmation emails through Postmark, and sync with POS systems, and Stripe takes deposits. Deploys run through a CI pipeline that copies the release to the servers, applies pending Rails migrations on the primary, then restarts the app servers one by one on the new release.
PagerDuty, Friday 12:07: [CRITICAL] api 5xx rate 41% (threshold 2%), p99 latency 30.0s. Restaurants are calling in saying the host stand app is frozen.
Probes left: 12 / 12 · Guesses left: 3 / 3
Logs you ask for open here, one tab each.
Try "Show me the database log". A log costs one probe.