Shawn Mervin

CloudWatch

AWS's monitoring service, and how a slowdown reaches my phone before it becomes an outage.

Visit aws.amazon.com/cloudwatchAWS

What it is

Amazon CloudWatch is AWS's monitoring and observability service. It collects metrics from AWS services and from your own code, stores logs, and raises alarms when a metric crosses a threshold. Lambda functions send their logs and metrics to it automatically, and an agent on a server can send memory, disk and swap figures.

Alarms can notify through SNS, which reaches email, phones and other services, or trigger automatic actions. Logs can be searched with Logs Insights, and dashboards put the important numbers on one screen. Retention settings decide how long logs are kept, which keeps the cost in check as a system grows.

How I use it

  • Seven alarms on mervinc.com: crashes, timeouts and errors per function, plus throttling on the agent and build crew. All feed one SNS topic that reaches my phone and inbox.
  • Error alerts are capped per function per hour, so one bad minute never becomes a hundred notifications.
  • Memory and swap alarms on the Rental Pilot API server email me before a slowdown becomes an outage.
  • The same alarms caught a WordPress login flood on the company websites.
  • Every Lambda function writes to its own log group.

CloudWatch is the right place for alarms on AWS because the metrics are already there. Crashes and timeouts never reach a function's own error handling, so something outside has to count them. I set alarms on the things that turn into outages and route them to my phone.

Questions

How do you know when a production system is in trouble?

CloudWatch alarms on memory, swap and function errors reach my phone and inbox, and a separate status probe checks every system every five minutes.

How do you avoid alert fatigue?

Error alerts are capped per function per hour, and alarms watch the few things that actually turn into outages.

Working with CloudWatch?

Tell me what you're building. Start a project or see the whole stack.

Got an operation that's outgrown its spreadsheets?

Tell me what your business does and where it's jamming up. If I'm not the right person for it, I'll say so and point you somewhere better.

What to send, and what happens next