[GUEST ACCESS MODE: Data is scrambled or limited to provide examples. Make requests using your API key to unlock full data. Check https://lunarcrush.ai/auth for authentication information.] [@astuyve](/creator/twitter/astuyve) "@ImSh4yy Maybe XX years ago. Cold starts and connection pooling stuff is mostly solved. The single-concurrency model is the original sin of Lambda and makes it totally cost-prohibitive at scale. GCP CloudRun and CF Workers got this right (in different ways) but also fucked it up too" [X Link](https://x.com/astuyve/status/1980777071367647397) [@astuyve](/creator/x/astuyve) 2025-10-21T23:23Z 8215 followers, XXX engagements "Managed services should not be free to reinterpret standards as important as HTTP just because they have a X pizza team. This is the second AWS service which incorrectly handles HTTP headers and even more annoyingly: - APIGW v1 uses case-sensitive headers - APIGW v2 does not" [X Link](https://x.com/astuyve/status/1981050432798470372) [@astuyve](/creator/x/astuyve) 2025-10-22T17:30Z 8215 followers, 9768 engagements "The AWS status page now says that recovery is progressing but there still seems to be widespread instability. Maybe we're seeing the beginning of a turn around" [X Link](https://x.com/astuyve/status/1980330436976079145) [@astuyve](/creator/x/astuyve) 2025-10-20T17:49Z 8191 followers, 1181 engagements "@gjergjiramku The control plane for AWS is a lot of Lambda Step Functions and dynamodb yeah. I'm sure there are good reinvent talks you could find where their engineers discuss this" [X Link](https://x.com/astuyve/status/1980339637869768716) [@astuyve](/creator/x/astuyve) 2025-10-20T18:25Z 8185 followers, XXX engagements "@JimsonRai @ImSh4yy Sorta The downside of this is that Lambda scales these functions down much faster than XX minutes if you look closely:" [X Link](https://x.com/astuyve/status/1980782119921348877) [@astuyve](/creator/x/astuyve) 2025-10-21T23:43Z 8185 followers, XX engagements "@ImSh4yy Not really they're pretty observable and understandable now. We also discovered the AWS pre-warms functions for you so most of the cold starts you see aren't even cold. eg:" [X Link](https://x.com/astuyve/status/1980782381499052156) [@astuyve](/creator/x/astuyve) 2025-10-21T23:44Z 8191 followers, XXX engagements "@ImSh4yy But you can also measure which parts of your codebase are actually causing the crushing delay and that gives you a ton of options. For example:" [X Link](https://x.com/astuyve/status/1980782549233463504) [@astuyve](/creator/x/astuyve) 2025-10-21T23:45Z 8190 followers, XXX engagements "@JimsonRai @ImSh4yy Good question I'll ask the node team" [X Link](https://x.com/astuyve/status/1980786004471538023) [@astuyve](/creator/x/astuyve) 2025-10-21T23:59Z 8184 followers, XXX engagements "@JimsonRai @ImSh4yy Also if you're using us already you can keep an eye on the "aws.lambda.ehanced.shutdowns" metric to track this. You can also look for the "aws.lambda.enhanced.invocations" metric and slice/dice your cold starts with either the cold_start tag or the proactive_initialization tag" [X Link](https://x.com/astuyve/status/1980788772146143388) [@astuyve](/creator/x/astuyve) 2025-10-22T00:10Z 8184 followers, XXX engagements "@aarondfrancis Give it AWS creds and make it give you a zip. I've done that in Lambda a few times back in the day" [X Link](https://x.com/astuyve/status/1981127507471716679) [@astuyve](/creator/x/astuyve) 2025-10-22T22:36Z 8215 followers, 1268 engagements "Here's my simplest explanation for the DynamoDB outage in us-east-1: X. The DNS Planner creates DNS plans for DDB periodically for the whole region. Then those plans are enacted by the DNS Enactor which runs one instance per AZ. X. Before an Enactor begins it checks that the plan it is going to apply is newer than the currently applied plan. It does this only once. X. One Enactor experienced unusual delays and took far longer than usual to go through each endpoint having to retry often for every endpoint. X. During this delay new plans were generated and new Enactors began applying them at" [X Link](https://x.com/astuyve/status/1981371507675701468) [@astuyve](/creator/x/astuyve) 2025-10-23T14:45Z 8215 followers, 16.3K engagements "I'm home with my sick kid who burst into tears because when I logged into Datadog the dog logo went away" [X Link](https://x.com/astuyve/status/1972691333669179736) [@astuyve](/creator/x/astuyve) 2025-09-29T15:53Z 8204 followers, 2534 engagements "I use Linux like an adult. Via ssh from a MacBook" [X Link](https://x.com/astuyve/status/1975956687128588794) [@astuyve](/creator/x/astuyve) 2025-10-08T16:09Z 8211 followers, 640.9K engagements "AWS suffered a massive outage beginning around 3am ET apparently caused by DynamoDB experiencing a DNS resolution failure and causing a total failure cascade in us-east-1. ec2 launch errors are still everywhere" [X Link](https://x.com/astuyve/status/1980238517788664288) [@astuyve](/creator/x/astuyve) 2025-10-20T11:43Z 8215 followers, 26.3K engagements "As AWS continues aggressive load shedding as they limp their systems back into a healthy state it'd be a good time to read the Metastable Failures in Distributed Systems paper. It's a quick read and I bet you'll find yourself nodding along:" [X Link](https://x.com/astuyve/status/1980343612669063290) [@astuyve](/creator/x/astuyve) 2025-10-20T18:41Z 8215 followers, 30.2K engagements "@apparentorder Lambda" [X Link](https://x.com/astuyve/status/1981093077667422445) [@astuyve](/creator/x/astuyve) 2025-10-22T20:19Z 8215 followers, XXX engagements "@morganfrnchstgg @apparentorder Yeah this outage could have just as easily been caused by an Executor that held the lock so long that the underlying DDB load balancers and hosts began hot spotting because the DNS records couldn't be rotated. It's foolish to conclude that "this would all be solved with locks"" [X Link](https://x.com/astuyve/status/1981404415450894573) [@astuyve](/creator/x/astuyve) 2025-10-23T16:56Z 8215 followers, XX engagements "@GergelyOrosz If I'm understanding things correctly the cleanup job which deleted the Slow Enactor's plan caused that plan itself to be an empty set. So the Slow Enactor was going endpoint by endpoint setting empty DNS records" [X Link](https://x.com/astuyve/status/1981418339248591005) [@astuyve](/creator/x/astuyve) 2025-10-23T17:51Z 8215 followers, XXX engagements
[GUEST ACCESS MODE: Data is scrambled or limited to provide examples. Make requests using your API key to unlock full data. Check https://lunarcrush.ai/auth for authentication information.]
@astuyve
"@ImSh4yy Maybe XX years ago. Cold starts and connection pooling stuff is mostly solved. The single-concurrency model is the original sin of Lambda and makes it totally cost-prohibitive at scale. GCP CloudRun and CF Workers got this right (in different ways) but also fucked it up too"
X Link @astuyve 2025-10-21T23:23Z 8215 followers, XXX engagements
"Managed services should not be free to reinterpret standards as important as HTTP just because they have a X pizza team. This is the second AWS service which incorrectly handles HTTP headers and even more annoyingly: - APIGW v1 uses case-sensitive headers - APIGW v2 does not"
X Link @astuyve 2025-10-22T17:30Z 8215 followers, 9768 engagements
"The AWS status page now says that recovery is progressing but there still seems to be widespread instability. Maybe we're seeing the beginning of a turn around"
X Link @astuyve 2025-10-20T17:49Z 8191 followers, 1181 engagements
"@gjergjiramku The control plane for AWS is a lot of Lambda Step Functions and dynamodb yeah. I'm sure there are good reinvent talks you could find where their engineers discuss this"
X Link @astuyve 2025-10-20T18:25Z 8185 followers, XXX engagements
"@JimsonRai @ImSh4yy Sorta The downside of this is that Lambda scales these functions down much faster than XX minutes if you look closely:"
X Link @astuyve 2025-10-21T23:43Z 8185 followers, XX engagements
"@ImSh4yy Not really they're pretty observable and understandable now. We also discovered the AWS pre-warms functions for you so most of the cold starts you see aren't even cold. eg:"
X Link @astuyve 2025-10-21T23:44Z 8191 followers, XXX engagements
"@ImSh4yy But you can also measure which parts of your codebase are actually causing the crushing delay and that gives you a ton of options. For example:"
X Link @astuyve 2025-10-21T23:45Z 8190 followers, XXX engagements
"@JimsonRai @ImSh4yy Good question I'll ask the node team"
X Link @astuyve 2025-10-21T23:59Z 8184 followers, XXX engagements
"@JimsonRai @ImSh4yy Also if you're using us already you can keep an eye on the "aws.lambda.ehanced.shutdowns" metric to track this. You can also look for the "aws.lambda.enhanced.invocations" metric and slice/dice your cold starts with either the cold_start tag or the proactive_initialization tag"
X Link @astuyve 2025-10-22T00:10Z 8184 followers, XXX engagements
"@aarondfrancis Give it AWS creds and make it give you a zip. I've done that in Lambda a few times back in the day"
X Link @astuyve 2025-10-22T22:36Z 8215 followers, 1268 engagements
"Here's my simplest explanation for the DynamoDB outage in us-east-1: X. The DNS Planner creates DNS plans for DDB periodically for the whole region. Then those plans are enacted by the DNS Enactor which runs one instance per AZ. X. Before an Enactor begins it checks that the plan it is going to apply is newer than the currently applied plan. It does this only once. X. One Enactor experienced unusual delays and took far longer than usual to go through each endpoint having to retry often for every endpoint. X. During this delay new plans were generated and new Enactors began applying them at"
X Link @astuyve 2025-10-23T14:45Z 8215 followers, 16.3K engagements
"I'm home with my sick kid who burst into tears because when I logged into Datadog the dog logo went away"
X Link @astuyve 2025-09-29T15:53Z 8204 followers, 2534 engagements
"I use Linux like an adult. Via ssh from a MacBook"
X Link @astuyve 2025-10-08T16:09Z 8211 followers, 640.9K engagements
"AWS suffered a massive outage beginning around 3am ET apparently caused by DynamoDB experiencing a DNS resolution failure and causing a total failure cascade in us-east-1. ec2 launch errors are still everywhere"
X Link @astuyve 2025-10-20T11:43Z 8215 followers, 26.3K engagements
"As AWS continues aggressive load shedding as they limp their systems back into a healthy state it'd be a good time to read the Metastable Failures in Distributed Systems paper. It's a quick read and I bet you'll find yourself nodding along:"
X Link @astuyve 2025-10-20T18:41Z 8215 followers, 30.2K engagements
"@apparentorder Lambda"
X Link @astuyve 2025-10-22T20:19Z 8215 followers, XXX engagements
"@morganfrnchstgg @apparentorder Yeah this outage could have just as easily been caused by an Executor that held the lock so long that the underlying DDB load balancers and hosts began hot spotting because the DNS records couldn't be rotated. It's foolish to conclude that "this would all be solved with locks""
X Link @astuyve 2025-10-23T16:56Z 8215 followers, XX engagements
"@GergelyOrosz If I'm understanding things correctly the cleanup job which deleted the Slow Enactor's plan caused that plan itself to be an empty set. So the Slow Enactor was going endpoint by endpoint setting empty DNS records"
X Link @astuyve 2025-10-23T17:51Z 8215 followers, XXX engagements
/creator/twitter::27568364/posts