Amazon puts all new hires on “on call” status for like a week every month
That’s insane. Where I worked you had to spend about 6 months learning enough that they trusted you to be on call. For months you’d just learn the systems. When you and your team agreed you were probably ready to be on-call, you’d be the “shadow” on call. The primary would get paged and you’d get paged too. You wouldn’t actually do anything, but you’d watch while the primary tried to solve the problem and take notes. If that went well it would switch to reverse-shadow. Then you were on call but there was an experienced person who was paged and ready to step in if you needed help. Only if that went well could you proceed to full solo on-call status.
being on call for stuff like this is pointless when you’re world wide and could literally just transition the stuff to a different team in some other part of the world
Where I worked there were 2 teams in 2 different time zones. But, you still were up late or early at times because there’s no perfectly-opposite time zone where team B is exactly 12 hours behind team A throughout the full year.
Also, if you recorded yourself doing on-call activities on YouTube or TikTok or something, you’d be fired. It would be the same thing as speaking to the press without authorization.
you should really watch some of these videos on youtube, there’s quite a few of them and yes you’re 100% spot on that if I recorded myself doing this stuff I would be fired but some of these kids go into great detail as to what issues Amazon is having, the details of said issues, and potential work-arounds/fixes their seniors suggest to them. When on call only the new hire is paged and it’s up to them to page a senior or someone else on the team when they’re stuck. The problem is these kids don’t want to admit they’re stuck to their seniors or other team members because they feel it’ll impact them negatively. They admit to it. So I’d say 8 times out of 10 the tickets they get paged for don’t get resolved and are passed on to another team in the morning. So the whole thing is pointless.
In one video I watched they do have a shadow but it was reversed. the senior is the shadow and ONLY during the day. the new grad hire is still doing all the work. and after office hours they’re no their own. I wish I could find the video again as it was awhile ago but one kid recorded himself working on a ticket at like 3am and I was almost screaming at the screen like “NO DON’T DO THAT OH MY GOD PLEASE CALL SOMEONE!”
It’s like when AWS went down a few weeks ago I was thinking “probably one of these new hires at 2am trying to fix something”
I just really hope that Amazon at least has it set up so that the really important stuff goes to actual, trained SREs. They could set it up so there are queues for things that aren’t business critical and have a very loose SLO that get assigned to the new grads. Or, the new grads get paged when the error rate for the service is 1% and if it gets above 3% someone who knows what they’re doing is woken up. If say all issues with Amazon’s Route 53 DNS service is shunted to new hires, AWS would be going down constantly.
That’s insane. Where I worked you had to spend about 6 months learning enough that they trusted you to be on call. For months you’d just learn the systems. When you and your team agreed you were probably ready to be on-call, you’d be the “shadow” on call. The primary would get paged and you’d get paged too. You wouldn’t actually do anything, but you’d watch while the primary tried to solve the problem and take notes. If that went well it would switch to reverse-shadow. Then you were on call but there was an experienced person who was paged and ready to step in if you needed help. Only if that went well could you proceed to full solo on-call status.
Where I worked there were 2 teams in 2 different time zones. But, you still were up late or early at times because there’s no perfectly-opposite time zone where team B is exactly 12 hours behind team A throughout the full year.
Also, if you recorded yourself doing on-call activities on YouTube or TikTok or something, you’d be fired. It would be the same thing as speaking to the press without authorization.
you should really watch some of these videos on youtube, there’s quite a few of them and yes you’re 100% spot on that if I recorded myself doing this stuff I would be fired but some of these kids go into great detail as to what issues Amazon is having, the details of said issues, and potential work-arounds/fixes their seniors suggest to them. When on call only the new hire is paged and it’s up to them to page a senior or someone else on the team when they’re stuck. The problem is these kids don’t want to admit they’re stuck to their seniors or other team members because they feel it’ll impact them negatively. They admit to it. So I’d say 8 times out of 10 the tickets they get paged for don’t get resolved and are passed on to another team in the morning. So the whole thing is pointless.
In one video I watched they do have a shadow but it was reversed. the senior is the shadow and ONLY during the day. the new grad hire is still doing all the work. and after office hours they’re no their own. I wish I could find the video again as it was awhile ago but one kid recorded himself working on a ticket at like 3am and I was almost screaming at the screen like “NO DON’T DO THAT OH MY GOD PLEASE CALL SOMEONE!”
It’s like when AWS went down a few weeks ago I was thinking “probably one of these new hires at 2am trying to fix something”
I just really hope that Amazon at least has it set up so that the really important stuff goes to actual, trained SREs. They could set it up so there are queues for things that aren’t business critical and have a very loose SLO that get assigned to the new grads. Or, the new grads get paged when the error rate for the service is 1% and if it gets above 3% someone who knows what they’re doing is woken up. If say all issues with Amazon’s Route 53 DNS service is shunted to new hires, AWS would be going down constantly.