Ben Kehoe
@ben11kehoe
Followers
17,317
Following
864
Media
1,952
Statuses
38,820
Siemens | Former vacuum salesman | AWS Serverless Hero | views my own
Nuremberg
Joined June 2015
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Texas
• 231106 Tweets
England
• 149844 Tweets
Mustafa Kemal
• 140900 Tweets
Notre Dame
• 82234 Tweets
MSNBC
• 78856 Tweets
Kentucky
• 33719 Tweets
Arkansas
• 29942 Tweets
Iowa
• 28972 Tweets
Sabalenka
• 26144 Tweets
Auburn
• 18339 Tweets
Marshall
• 17620 Tweets
#AEWAllOut
• 16321 Tweets
#الهلاليين_يرفضون_لودي
• 15083 Tweets
Pegula
• 14066 Tweets
T-岡田
• 11854 Tweets
Ozee
• 11791 Tweets
#UFCVegas97
• 10841 Tweets
#ボクらの時代
• 10462 Tweets
Go Green
• 10311 Tweets
Pinned Tweet
Transition to the
#cloud
: treat servers like cattle, not pets. Transition to
#serverless
cloud architecture: treat servers like roaches
25
119
427
Living the serverless dream: Christmas Day 2021, when everybody who has bought a Roomba since Black Friday opens them in about a four hour window on Christmas morning, was completely hands off keyboard for us. Literally nothing needed to handle the massive influx of traffic.
17
122
1K
Reminder that the best way to test your AWS configuration and credentials are correct is:
aws sts get-caller-identity
(NOT aws s3 ls)
GetCallerIdentity tells you what account and user/role you are, and requires no permissions so will always work
13
71
500
GCP having availability zones in purely software terms and not as physical separation feels like a very Google thing to do
12
34
500
#awswishlist
CloudFormation should accept a blurry photo of an architecture diagram I drew in crayon on a McDonalds placemat in addition to templates in YAML and JSON
25
34
432
Well, this is certainly a thing that is happening
59
36
423
So much going on here but...Friday was my last day at iRobot. It's been a good run, but I've decided to take some time off to search for what's next for me. I think it's unlikely to be robotics/IoT, as what I do (cloud-wise) isn't confined to a particular vertical.
74
9
410
In case you need an SCP to prevent people from creating open Lambda URLs, it's this:
7
69
403
This is HUGE: you can now execute queries against Aurora through HTTP, basically eliminating the friction of using it with Lambda. This opens up a whole new world of serverless data modeling on AWS
9
126
383
Repeat after me: S3 is not a filesystem. S3 objects are not files. I will not think about S3 semantics in terms of files.
AWS just released mountpoint - an open ssource high-throughput file client for S3. Written in Rust!
11
125
570
21
52
379
I trust AWS IAM to secure my applications. I don’t trust the IAM docs to tell me how.
13
75
352
please, I'm begging you, stop stuffing temporary credentials into ~/.aws/credentials 😡😭
27
37
327
Christmas Day operations at iRobot so far: requested a limit increase for a firehose stream. That’s about it.
Anyone who says
#serverless
isn’t ready for production doesn’t know what they are talking about.
#HugOps
to anyone out there whose system is struggling today.
7
67
306
I'm still working with my team to mitigate the after effects of the day-long AWS outage yesterday, including dealing with follow-on AWS issues. I've gotten three hours of sleep and it's ruining my Thanksgiving day.
Hot take: I am thankful we have built serverless on AWS.
16
8
306
Some news: next year I'll be starting a job at
@Siemens
Digital Industries in Erlangen, Germany! Looking forward to many interesting challenges to bring the serverless mindset to. And I'm going to be an EMEA Hero!
62
2
262
I am wholly against this mindset. It downplays the operations and maintenance burden assumed by building something yourself. Looking at TCO, it's almost always a better option to accept the 80% fully-managed-service solution than take on all the extra burden of a custom build
20
70
231
Sure, AWS Lambda is cgi-bin...on a fleet of servers that scales for you, with an OS you never need to patch, with no network attack surface, where each process gets its own entire compute env, billed only for the milliseconds it runs. The execution model isn't the new thing.
8
67
224
Our newer models don’t suffer from this, but it took YEARS to figure out how to make the sensor robust to this while remaining cost effective. The constraints on consumer electronics are brutal
@hardmaru
I wish they had also created a diverse dataset of rugs so that it didn’t confuse black stripes with cliffs and I could finally get my entire house cleaned 😂
110
1K
5K
12
25
219
Roles Anywhere is a huge deal. It will be the key enabler of removing long-term creds from their last bastion: on-prem servers with no other source of identity. The key is "will be", because what's there now is only something you want to experiment with 1/
7
53
215
Stop putting AWS temporary credentials in ~/.aws/credentials or environment variables! There's a better way. I wrote about the options for AWS configuration:
3
33
211
“1500 lines of CloudFormation became 14 lines of CDK” It’s important to understand that the deployed application still has 1500 lines’ worth of operations and maintenance ownership, not 14
11
26
212
@bryanl
In-app purchases. They are designed to prey on people with addictive personalities. Imagine if the very public discourse around hooking and squeezing money out of "whales" took place (as publicly) in the gambling industry.
2
8
191
I'm always reluctant to toot my own horn, but if you work with AWS in Python and you do role assumption without aws-assume-role-lib, you're missing out.
5
23
182
Periodic reminder for a CloudWatch Logs protip: replace newlines with carriage returns ('\r') in your multi-line log messages (e.g., stack traces), and they will stay as a single log entry, but be viewable as multi-line in the log stream.
7
37
180
So, this is exciting for me: I have a post on the AWS Blog!! How to use Step Functions for approval emails, using my SAR app:
3
44
177
Why can’t things in Apple Wallet expire themselves, in particular airline boarding passes?
20
10
175
These CIO are making a bad decision, but if you are selling to them, there is a lot of profit to be made off it
20
25
178
Why don't any of the characters in Dune starve on that planet?
Because of all the sand which is there
21
19
173
I see a lot of people who aren’t fully aware of the difference between ~/.aws/config and ~/.aws/credentials. While it is more or less fine to just use one or the other, I think it’s worth understanding their intended purpose. So here’s a short explainer.
5
53
172
Some days I feel like the ship, some days I feel like the excavator.
8
32
171
If you want an idea of the challenges of consumer IoT: we stopped making robots with firmware that talked to our pre-AWS system in July 2017. Last month, for the first time, we went a full day without a robot with that firmware newly coming online.
7
20
165
Great news! As of boto3 version 1.29.1, API schemas are stored compressed, reducing the package size (of botocore) from 85 MB to 20.
8
16
162
A thing I'd really like to see is an acknowledgment that "if that were true, nobody would work for us" was also false—that they know many many people do not have the security to leave a job due to labor and economic conditions, and thus have severely reduced bargaining power
3
29
157
Without a job, this AWS outage isn't stressful for me at all. I'm not sure it can't work as a general outage strategy, but something to consider
10
11
162
I've seen a lot of AWS newcomers get a bit confused about what identities they are using to access AWS services, and have seen notions of "logging in to AWS" that are a bit off base. So I wrote up how principals (Users and Roles) in IAM work!
@AWSIdentity
4
32
159
True customer obsession is telling them about good features from your competitors.
Amazon's new S3 Object Lambda is pretty dope. You can use a Lambda function to process data as it's being retrieved from S3. What an elegant way to extend a managed service.
20
163
1K
3
16
154
It's disappointing to see this from AWS, the company that's famously API-first and famously bad at 1st party UIs, and that has not delivered a significant improvement in session catalog UX in the 8 years I've been going to re:Invent
I'm sad to share that AWS is reaching out to all third-party Re:Invent session trackers with a takedown notice. I have no choice but to comply.
It wouldn't be so bad if the official catalog had a better UX. I'm especially going to miss
@RaphaelManke
's calendar browser.
33
12
190
8
19
147
If you use boto3 and don't know about aws-error-utils, allow me a shameless plug that you should check it out. boto3 errors are cumbersome, and it makes them easier
8
33
140
Fargate is a bigger deal than EKS. This is Docker like EC2. Start, Stop. No cluster admin at all! It is NOT serverless. Your containers are probably servers
#reInvent
8
70
140
Every high-privilege or admin role should be paired with a read-only version. The ability to go into an sensitive environment with the confidence that you *can't* modify it is super useful
10
16
136
I realize I'll never win this battle, but we should stop calling it "infrastructure". It's no longer "infra" (below). It's not something your application is deployed "onto". You have a graph of resources, some (as few as possible) of which have custom code attached.
20
16
141
To give you an idea of what the spike is like, here's a graph from a previous Christmas that we've shared in the past:
1
5
141
IAM policy details are far too complicated for static tables in the docs. AWS should maintain a proper database of the info, including resource policies as well, with a browsable and queryable web UI as well as an API.
#awswishlist
6
23
139
The thing to remember about Lambda container support: when you use a container, you own the entire image. You're responsible for all updates to that image. When you can, use zips. Own less stuff, focus better.
#reInvent
4
17
136
For any cloud engineer that hates “PIN number” and the like, I’d like to point out that
@awscloud
service endpoints are all subdomains of . Amazon Amazon Web Services.
You’re welcome.
7
29
132
People are going to end up with a giant mess of Function URLs and then put CloudFront as a routing layer in front of them and slowly reinvent for themselves a poorly-implemented subset of API Gateway functionality
10
8
131
An AWS developer pointed out to me today that CloudFormation-first development for AWS services is like mobile-first web development: you should do it because it's not something you can expect to get right if you try to include it later on in the process
9
14
127
Adding to my long list of ways I am Not Fun, I worry about the people new to AWS who are not in on the Infinidash joke.
29
3
129
The most bizarre thing I've ever encountered in AWS, by a wide margin: when you're subscribed to Shield Advanced, every month it randomly selects an account in your Organization to bill. IT CHANGES EVERY MONTH. I'm at a loss to even imagine how that even happens!
24
15
129
I heard a rumor that the AWSConfigRole managed policy is getting deprecated, but I've only received 145 emails about it, and I have more accounts than that, so I'm not sure whether it's true or not.
11
7
123
I'd like to note that this is only possible because of the operational excellence and the operational work being done by AWS service teams—and as we use several dozen services to build this, it's a lot of teams! They're doing the work, so we don't have to.
1
2
125
If you have a container that is active when it is not handling data, it👏is👏a👏server👏
6
61
122
.
@QuinnyPig
's piece on the damage done to the trust we can have in AWS is thorough, thoughtful, and accurate. I agree with it completely, and I'm also angry about how the trust that so many Amazonians have worked hard to build has been undermined.
3
37
120
CloudFormation is an infrastructure graph management service, and needs to act more like it. This is critical for the future of deployment for
#serverless
architecture
6
36
120
Move your dev loop closer to the cloud. This is the future, not locally mocked services.
Have you heard about
#AWSSAM
Accelerate yet? synchronize your
#serverless
app to an AWS developer account..in a couple of seconds, it's like coding local.... in the cloud. 👀🤯
7
42
153
5
16
122
"We need a multi-cloud strategy"
give me a horror story from your specialty in five words or less
18K
863
9K
3
10
118
"Surely We Can Do Better Than Elon Musk" is such a good takedown of what's wrong with society's attitude towards Elon Musk
1
41
119
CloudFormation should not allow any service teams to ship new resources to have properties that are stringified JSON, even if the APIs that back them require it; the property should be JSON and the resource provider should do the serialization
11
3
119
@editingemily
A problematic part of the "new kingmakers" theme is "devs should be able to do whatever they want and refuse to do anything they don't want to do, and everyone else has to deal with the fallout"
8
10
119
The thing that hurts most about the "serverless" label getting slapped on everything is hearing someone say "we used XYZ Serverless—but remember to shut it down after your experiment, or you will be surprised by your bill". Antithetical to what serverless is about
11
11
115
#reinvent
tip: there's no need to go to everything. You don't need to prove or show off how much energy you have. Take breaks, get sleep, make sure you take care of yourself.
4
23
108
Being well-versed in IAM is an AWS superpower. Training up on it is well-worth your time.
👉Being comfortable with AWS IAM is the first step in securing 🔑 your resources on
#AWS
. Check out this course that explains the concepts to help you get there! Spread the word, encourage your colleagues to enroll, and get started on
#AWSIAM
!
@AWSIdentity
3
33
113
5
12
112
This is going to get buried under re:Invent news so I'll bring it up again in January, but I've been building tools to make working with AWS SSO easier.
4
26
106
"IAM users" should be renamed "IAM long-lived access keys". These long-lived access keys are needed for automated processes running outside AWS that don't have a source of identity that AWS understands. They are no longer needed—or indeed recommended by AWS—for actual users.
6
10
108
#awswishlist
the IAM role and policy creation wizards in the console should give you CloudFormation snippets to copy (before the resource is created, so you don't actually need to create it). These are some of the most annoying CFN resources to write manually.
9
7
103
@jessfraz
@kelseyhightower
So,
@iRobot
has proper scale, and we have no VMs or containers in our OLTP system, and for analytics no VMs and containers only in AWS Batch. We're a little different as we built this up greenfield in 2015-16 & had no legacy infra. Internal infrastructure has VMs (eg Jenkins).
3
15
105
I like this diagram. A big part of it comes from projecting the collective knowledge of everyone else onto each of them individually!
How to kick imposter syndrome to the curb with
@ben11kehoe
, live from
#reInvent
.
@fayecloudguru
#AWSreInvent
0
2
23
2
14
108
A game-changing update from
@AWSIdentity
today. IAM Permissions boundaries let you control what permissions a user can delegate to an IAM role they create. Details and caveats follow...
3
54
103
@QuinnyPig
@awscloud
Exactly, that is why they have been trying to hire someone who knows how to sort!
0
0
103
Azure really seems to be failing at the basics of multitenant security. I have a hard time wrapping my mind around what the process and culture must be that wouldn't catch this at the design phase, let alone allow it to reach customers.
The port, 40008? Why would someone choose such a random port?
Well..
Because other ports were taken. By other customers on the same machine!
You could just upload a script that tries every port in the environment and steal tokens belonging to other customers 😳
(5/10)
2
16
237
8
12
105
CloudFormation modules are the first step in bringing real levels of abstraction into CloudFormation proper. While they can't be today, CDK constructs should end up as modules, so your abstractions don't disappear during `cdk synth`
13
17
102
AWS has a commitment to train 29 million devs by 2025. That’s a very specific number—why not 30?
24
7
100
Remember that monitoring is work, and observability is key to reducing operational tasks
I do want to clarify: it's still not zero operational burden! I don't find "NoOps" to be a useful concept. Even though we didn't need to touch the system, we still had to monitor it to *know* that nothing needs tweaking, and that's real work.
2
5
62
3
10
100
I've been explaining the AWS SSO client login process a lot (the one the AWS CLI v2 uses), and somebody asked me to make a sequence diagram, which was a good idea. So here it is:
3
18
98
Google constantly decommissions popular products because they are trying to teach the world that attachment is suffering.
8
19
96
🚨🚨🚨AWS SSO support has finally landed in the CDK in v2.18.0 and v1.150.0🚨🚨🚨
1
21
98
I wanted to be writing about cross-account role trust policies, and how credential_process means you should never engage in stuffing creds into ~/.aws/credentials, and how cdk synth should be deprecated, but here I am, writing about how IAM documentation is inadequate 😢
8
7
94
The struggle of being an AWS customer, summed up in a single sentence: "I’m a little worried that there are multiple competing solutions in AWS and not a clear strategy"
@ben11kehoe
@salman_paracha
@forrestbrazeal
The Cloud is an operating system and it needs a thought out package management/App Store experience that supports both creators (software companies) and cloud users. I’m a little worried that there are multiple competing solutions in AWS and not a clear strategy
2
1
23
8
13
95
Every AWS VP/GM needs to build a solution that spans their product AND products from several other AWS orgs. AWS sets up teams to be autonomous & customer-obsessed, but those teams too often miss how their piece fits into the bigger puzzles their customers are putting together
Unpopular opinion: every AWS VP/GM should build a solution on their product. Once a month.
17
6
179
5
12
92
Is there a hashtag for “in my hotel room every night by 8 and asleep by 10”?
3 days to go! Which re:Invent after hours event are you most excited for? Reply with
#reInvent
&:
#rePlay
#MidnightMadness
#BoardGameNight
#BingoNight
#PingPong
#MovieNight
#80sSingAlong
#Dodgeball
#Tatonka
#ArtsandCrafts
#RestaurantReceptions
14
14
46
17
2
93
I would flip this around: IAM users should be *primarily* for long-lived access keys for non-humans. Avoid using IAM users for humans if you are not already using them. Instead, federate your existing identity provider or use AWS SSO with its built-in identity store.
[054/100] IAM users aren't always for human beings. Sometimes you'll use them for programmatic (service) access, or to feed your dog with a string of cloud-connected IOT devices.
#100DaysOfCode
#100DaysOfCloud
0
4
12
11
19
90
Well well well, if it isn't Mr. Privatize The Gains here to socialize the losses
Where is Powell? Where is Yellen? Stop this crisis NOW. Announce that all depositors will be safe. Place SVB with a Top 4 bank. Do this before Monday open or there will be contagion and the crisis will spread.
5K
2K
15K
2
7
91
The next step beyond this is the Lambda code running inside S3 itself (probably called Lambda
@S3
?) Computeful state is the future.
9
17
89
Not AI, not a robot 🙄🙄🙄🤦♂️
14
7
87