Lies AWS Told Me

This is about Availability Zones.

You probably have seen the AWS status page. It's usually a sea of green checkmarks designed in the 90's dot-com bubble.

It's become a bit of a meme. Even when half the internet is down, you might only find a yellow icon hiding somewhere in that list.

But that's not the lie we're talking about today.

My AZ is not your AZ

If an incident is reported in the status page, it might be specific to an Availability Zone, such as us-east-1a.

Here's the issue: Your us-east-1a might not be my us-east-1a.

Popular Lore of AWS™ states that too many people were using the first few AZ's within a given region. This resulted in AWS running out of capacity on popular server types. (Yes, that's totally possible - that's why Capacity Reservations exist).

To combat this, AWS decided to do some "load balancing" - each AWS account's us-east-1a, us-east-1b, us-east-1c (and so on, for each region) can be assigned a different "actual" availability zone (e.g. a different physical building).

Where's my stuff?

You can confirm which Availability Zone your AWS accounts are using under the hood using the AWS CLI:

1# List AZ's in your acount for a given region
2aws --profile some-account --region us-east-1 \
3    ec2 describe-availability-zones

You're looking for the ZoneId parameter, which will have a value that looks like: use1-az1.

Here's what I see across 2 different AWS accounts for the US-East-1 region:

Here's what you're looking at:

• us-east-1a is use1-az6 in one account, and use1-az4 in another account
• us-east-1b is use1-az1 in one account, and use1-az6 in another account
• (and so on)

We can see that us-east-1a for left-hand account is in the same physical location as us-east-1b for the right-hand account.

The AWS Status Page

When AWS has an incident, the status page updates don't always say which actual AZ is having an issue. (Sometimes the issue is region-wide in which case this is a moot point).

This is problematic for obvious reasons - if the trouble is in us-east-1a, you don't actually know if your us-east-1a is the one effected!

Here we see that the alert that does include the real AZ:

Pro Tip

The status page has RSS feeds for each service within each region. However, the One True RSS Feed™ is a bit hidden. Here it is, for your benefit: https://status.aws.amazon.com/rss/all.rss

I use the RSS feed Slack app to pipe RSS feed updates into a #monitoring channel. It's not super noisy, and it's 100% better than looking the AWS status page.