After 10 years as an AWS customer and open-source contributor, they deleted my account and all data with zero warning. Here's how AWS's 'verification' process became a digital execution, and why you should never trust cloud providers with your only copy of anything.
This shouldn’t happen and the OOP clearly knows what he was doing but putting everything in a single provider with multiple services clearly is not redundancy.
The author put it well:
Multi-region cloud computing is already difficult and expensive enough, multi-cloud is not only technically complex but financially and legally fraught with uncertainties. At that point you’re giving up so much of the promise of cloud computing that you might as well rent rack space somewhere, install bare-metal infra, and pay someone to drive there to manually backup to tape every 3 months.
This level of technical purity is economically unfeasible for virtually everyone, that’s the whole point of paying a vendor to deal with it for us. And you know who doesn’t need to put up with the insane overhead of multi-cloud setups? That’s right, Amazon, Microsoft, and Google, who will be getting paid for hosting everyone else’s multi-cloud setups while they get to run their huge infra on their own cloud without fear. The last thing GAFAM competitors - especially OSS projects - need is even fewer economies of scale.
Stop with the victim-blaming, this blunder is squarely on AWS.
Yeah this is the right take, IMO.
Good on you to those of you who actually do multi-cloud backups.
Even if this was just a loss of their infrastructure, it would be catastrophic to any company without good infrastructure as code practices.
Not to mention the downtime.
This sure as shit doesn’t look “customer obsessed”.
Lol “victim blaming” used in the context of a situation with no victims is crazy. The author signed a contract that allowed this outcome and is SHOCKED it happened to them. You and their naive asses can enjoy blaming everyone but yourself.
Did you even read the article? Even under the VERY GENEROUS interpretation of contract law that contracts can’t be predatory (which is not a particularly popular philosophical stance outside of cyberpunk fiction), AWS MENA fell short of even their typical termination procedures because they accidentally nuked it while doing a dry-run.
I don’t know where you work but if we did that to a paying customer, even IF there was a technicality through which we could deny responsibility, we would be trying to make it right.
Multi-cloud is a significant amount of effort to pull off.
Being on one cloud provider across multiple regions is often plenty of redundancy.
Being available across multiple cloud providers is really REALLY difficult
Multi-cloud is difficult, that’s true. But having backups outside the single cloud is easy.
That way if your cloud provider pulls the plug, you will have to reconfigure everything but at least your data stays intact.
To be able to recover from something like that you don’t need multiple working cloud setups. You just need backups, so that in an event like OOP’s, you spend a few weeks rebuilding the configurations instead of years rebuilding your projects.
It really depends, pulling hundreds of GiB out of AWS for backing up on say GCS is going to add up extremely quickly. The cloud companies make it intentionally painful to leave or interop.
Even large projects rarely have hundreds of GB of code. They might have hundreds of gigs of artifacts and history, but not all of that needs to be backed up. That’s where tiered backup strategies come into play.
Code (or what ever else is the most painful to recover) is backed up in e.g. git, with version history and many different locations.
Artifacts either don’t need a backup at all, or maybe one copy. If they get lost, they can be rebuilt.
Temporary stuff like build caches don’t need backups.
You don’t even need to backup the VMs. Backing up a setup script is enough. Sure, all of this is more complicated than to just backup your whole cloud storage space, but it also requires orders of magnitude less storage.
In this guy’s specific case, it may be financially feasible to back up onto other cloud solutions, for the reasons you stated.
However public cloud is used for a ton of different things. If you have 4TiB of data in Glacier, you will be paying through the absolute nose pulling that data down into another cloud; highway robbery prices.
Further as soon as you talk about something more than just code (say: UGC, assets, databases) the amount of data needing to be “egressed” from the cloud balloons, as does the price.
Retrofitting stuff is of course difficult. If it’s done from the beginning it wouldn’t be that difficult or expensive.
4TB isn’t that much. That’s small enough that it can fit in a cold backup on a hard drive or two.
Multi-cloud is far from trivial, which is why most companies… don’t.
Even if you are multi-cloud, you will be egressing data from one platform to another and racking up large bills (imagine putting CloudFront in front of a GCS endpoint lmao), you are incentivized to stick on a single platform. I don’t blame anyone for being single-cloud with the barriers they put up, and how difficult maintaining your own infrastructure is.
Once you get large enough to afford tape libraries then yeah having your own offsite for large backups makes a lot of sense, but otherwise the convenience and reliability (when AWS isn’t nuking your account) of managed storage is hard to beat — cold HDDs are not great, and m-disc is pricey.
In terms of service availability, yes.
In terms of data backup, no.
Yeah, but he was paying the provider for providing the redundancy on this level.
We shouldn’t blame him on the technical level for what is a problem with the provider on the organizational level. If this had happened to a Fortune 500 company, Amazon would have had an army of lawyers descend on them.
“i didn’t put all my eggs in one basket, i put them multiple metal boxes… that i paid the same guy to hold for me. it’s different.”
During the entire article I kept thinking how it wasn’t best practice to use the same cloud for everything, that you needed a different cloud because occasionally, your account can just get nuked from orbit. People dont understand that these corporations are really just held together with luck prayers and duct tape. Software enables us to develop in the worst ways possible, and then call it production.