Archive Are We Down?

How to play

Something is wrong in production. You get the system and the first alert; find the root cause.

A new incident every day.

Bookings crawl as the day goes on

Wayfarer is a hotel and holiday-rental booking site, mostly Italy and France, on AWS. Browsers and apps call a Node.js API behind an Application Load Balancer; the API fleet is EC2 instances in an Auto Scaling group with a target-tracking policy (average CPU 55%, min 6, max 40), launched from a launch template pinned to a baked AMI. Deploys copy new code onto running instances. Search runs on a managed OpenSearch domain; a separate pricing service runs on ECS. Data: Aurora MySQL (writer plus reader) and ElastiCache Redis; static assets from CloudFront. Availability comes from hotel partners' APIs. Payments go through Adyen, booking emails through Amazon SES, and DNS is on Route 53. A shared tooling account runs nightly cleanup Lambdas (old snapshots, unused AMIs, stale log groups).

ticket

Friday 17:20 CET, support escalation: 'the site is crawling'. Customers say search results take 6-8 seconds, the booking button spins, and it has been getting worse since lunch. A few bookings time out; most eventually go through.

−$0lost so far · −$3,200/min · 0:00

Probes left: 12 / 12 · Guesses left: 3 / 3

Logs you ask for open here, one tab each.
Try "Show me the database log". A log costs one probe.