I love GCP but honestly have no idea how these big internet properties can make the $ case for “the cloud”, unless they’re also planning to fire 90% of the Ops/SysAdmin team (ala Evernote).
We run on bare metal and to move to GCP would triple our hosting cost, not including bandwidth (which is free* for us).
Do people get discounts for doing PR? I’m really curious.
The hardware and bandwidth costs are only small parts of the full cost of running your own infrastructure. There are so many other benefits:
- you don't have to support your own low level services like Pub/Sub, Cloud Storage, Spanner, etc.
- you don't have to worry as much about things like HIPAA compliance
- developers don't have to wait for you to allocate more hardware, greatly slowing down the software development process
- it's much easier to test your applications, since you can spin up a full copy using rented infrastructure, and then throw it all away once your tests are complete
- many other benefits that I'm forgetting
And of course if you buy enough from them, you do get additional discounts.
For many (most? I don't have data) companies, the actual cost of the infrastructure and bandwidth itself is tiny compared to all the other "hidden costs".
So having experienced Ops in both cloud and non cloud companies. A few rebuttals:
- Bandwidth is VERY expensive in AWS compared to what you can buy in Datacenters. A 1 GBPS connection is so "cheap" you can get it and a 42U rack at HE.net for $400/mo ( http://he.net/colocation.html ). That same 1GBPS is roughly 300TB/month. Assuming it's all outbound - that's roughly $21k in AWS.
So bandwidth is a small cost IF you're not moving large amounts of data out of AWS. If you stream media - it can be a HUGE portion of your bill.
- HIPAA Compliance - That and other compliance aspects assumes that you need those compliance frameworks. In enterprise world that might be the case (SOC2 and ISO27k1 are very common requirements for Enterprise SaaS clients). However in the B2C? Users don't care.
- Waiting for hardware allocation. Once you get to a certain size you have hardware spare, typically. Developers can use that for their tests. You can also have your datacenter/operations group work with your developers during project planning so the code and HW can be ready at the same time.
That being said, I don't disagree there are advantages to the cloud. I'm AWS certified, I drink the coolaid. But saying that there are "so many other benefits" is ignoring the fact that there are "so many benefits" to on prem too. There are pro's and cons to both sides.
> Waiting for hardware allocation. Once you get to a certain size you have hardware spare
Requisitions for non-cloud server resources can takes hours, days, and in some places weeks; even when there is availability depending on the org. At best it takes close to an hour compared to a few seconds in the cloud via a web dashboard.
First of all, as the sibling comment mentioned, no one is waiting for weeks for hardware requisition. Everyone is using virtualized environments and have capacity to spare.
Spinning up a new instance in VMware is not harder (in fact easier) than in AWS and hardware costs are SO much cheaper than Amazon's margins.
Furthermore, If you have spiking workloads or hyper growth and can't do proper capacity planning - that's exactly the original AWS value prop, run a hybrid private/cloud environment and extend to the cloud when needed.
I beg to differ, I am a systems engineer for a Fortune 500 retailer, and my team has been waiting weeks, and have been told it will be some time next year, for some vmware instances we need.
That being said, not every organization is as dysfunctional.
I think the best approach would be a hybrid one, where developers could spin up instances in the "cloud", and ops would migrate them into the main infrastructure when it made sense to do so.
I'm glad that I'm not alone in that experience (albeit from the development side). A two week turnaround for a development VM would be amazing; on average, we're closer to a month from request to logging in.
I've actually seen some customers span their openstack cluster across on prem & gcp. Giving them the ability to dynamically scale new hypervisor nodes on GCP.
This works great for gaming companies who can see 10x loads during the first few weeks/months of a new video game release.
Several of my family members and relatives who work in big corporations have waited months for a new server. It is more common than you think, and often it has nothing to do with physical capacity but rather bureaucracy and compliance overhead.
I guess you've never worked for a large enterprise? The amount of beaurocracy and time wasting is staggering - where I work it takes days at a minimum to get a new VM up, but usually weeks.
> Everyone is using virtualized environments and have capacity to spare.
I don't understand this argument. Startups usually do not have money to have physical servers sitting around just waiting to be utilized, not to mention the IT resources to stand up those servers just in case someone might need them later. You're saying that everyone has a ton of servers sitting in the rack just waiting to have containers deployed to them? Who?
It's about bin packing. Underutilized servers can have more VMs packed into them. Not physical servers sitting around unused, but additional capacity on in-use machines.
you can do `bin packing` with k8s as well. Allowing you to have a best of both worlds.
From k8s webpage (https://kubernetes.io/): "Automatically places containers based on their resource requirements and other constraints, while not sacrificing availability. Mix critical and best-effort workloads in order to drive up utilization and save even more resources."
Bin packing is the actual academic term for this class of problems so the intended snark in using quotes is a bit misplaced.
Furthermore, running k8s to pack containers into AWS instances that are are themselves actually full blown VMs running on a hypervisor is of course one way to do computing. It is perhaps worth considering that there are others, more direct ways that can be achieved more economically.
You can use virtualization on prem; it’s trivial. No one is still racking hardware on demand. Don’t want to use Xen or KVM? Kubernetes is your solution if you’re a containers shop.
On prem delivery/provisioning is not slower when executed properly.
Could be wrong but I don't think virtualization or container technology is the blocker. It's the software that keeps track of who's allowed to use it and who's currently using it that's the problem. With AWS it's easy, because you pay money for the privilege. It's harder for large orgs, when divisions and teams don't directly pay to access servers. 3rd party clouds solve the issue when teams are given budgets to use them.
Yeah I definitely agree there are pros and cons on both sides. Every company is different, etc. What I wanted to point out is mainly that it's naive to think the costs of hardware and bandwidth are always most significant or even in the top 5 most significant factors behind whether or not a company should use the cloud. It's a frequent misunderstanding I see posted both on Reddit and hackernews.
GCP doesn't have a separate HIPAA cloud. GCP went through the trouble to certify the services themselves for HIPAA. That means you don't need to wait for us to update our `HIPAA` cloud. You are always getting the latest releases.
Here's a full list of our certifications. We're rapidly adding more. Feel free to email me at bookman@google.com if you have any questions:
https://cloud.google.com/security/compliance
I don't have very detailed knowledge, but my impression is that this move doesn't make long-term financial sense for the company, but it's being pushed by activist investors / board members who want to appear like they're good stewards of money because investors think cloud is cheaper / etc. (Note that this is on investors.etsy.com, not their tech blog; you can also find a call for public cloud in the Black & White Capital letter, in the NYTimes story from a couple weeks ago, etc.) I heard some rumors about this when I was applying for a job right before the layoffs happened, and I didn't hear anyone expressing interest in moving out of their datacenters before the layoffs.
Yeah. The exact same thing happened at Evernote. They hit a sales slump, new “corporate” management, cuts etc -> GCP.
The Ops people there were pretty proud of their infrastructure, but they took it to a pretty extreme level. I think they were racking their own stuff and even running their own routers etc.
Racking your own servers and running your own routers is extreme? I don't know about you, but I haven't set foot in the DC in two years after initial setup.
The investor public market bit could be true but it's not about whether cloud is cheaper but how it is treated from a financial reporting perspective.
If you own the hardware or use debt it appears on side of the balance sheet where as if you just pay a provider then it is a cost of business like any other software license so it makes the company look finally better even if the overall actual cash cost is 10-20% more.
Also Etsy has had all Senior Management replaced so new technology leader could just be leading the charge as Etsy was a bit of an anomaly running their own infrastructure because that isn't really a competitive advantage for them from the business customer side.
To answer your question, people get discounts just for negotiating, no PR required. I would guess the conversation went something like this: "___ is how much we pay to host Etsy on bare metal right now. Can you match that?" "Yes."
I'm oversimplifying - there are a bunch of other considerations, like guarantees about future prices and capacity availability and whether having more flexibility makes it worthwhile for Etsy to pay more, but fundamentally Etsy looked at their current costs and asked Google to make an offer. (I would assume they actually had all three major players bid, and Google won.)
Also to your point, the announcement specifically called out Google's AI and big data capabilities. Those are workloads that can involve a lot of bursting, so the cloud might be especially attractive. Imagine trying to implement BigQuery in bare metal - you can do it, but you have to have a lot of extra servers lying around just in case someone tries to run an analytics query. The same goes even more strongly for something like a model training workload. For bursty enough workloads, running in the cloud can be cheaper than bare metal even given the cloud provider's profit margin.
Depending on how much model training Etsy wants to do, something like TPUs might have also given Google a fundamental edge in costs over bare metal or other providers. I don't know enough to know exactly what factors went into Etsy's decision, but I think these are reasonable ideas.
Okay, that makes a lot of sense. If Google told me they’d match my pricing then sure, I’d probably look at doing that too.
Still a lot of issues on GCP for low latency DB deployments though (no permanent NVME, higher latency on the network backed block storage, etc).
Presumably you start looking at Spanner but that’s it’s own world of hurt (higher latency than MySQL/Postgres, a reeaaallyy basic query planner, the list goes on). You have to change the way you think about data with it and pretty much re-architect your data access.
I know for a fact that AWS that AWS negotiates. My buddy used to work at Netflix and he told me that they got a really good rate for AWS that was not available to the general public.
They might not bid in the strictest sense of the word, but when the vendors are mostly 3 players (Google, Amazon, and Microsoft) you just ask each one for a quote. I don't see how functionally that is different from bidding.
Btw, I know there are other hosting providers out there (Rackspace comes to mind).
I can't think of any one of them that would be able to deal with Netflix scale without shitting the bed, missing features, or an almost unacceptable lead time as they build out infrastructure.
As with everything else, the prices will reduce over time. But to be honest: you are not the market. If you have a functioning, reliable infrastructure that isn't too expensive or PITA to maintain, you have a good enough infra and should continue using that.
If you're a startup who wants to minimize capex and quickly spin up a reliable, secure infra to host your app on to get it out of the door ASAP, you would go with cloud.
If you're a non-technical company with a small IT department that wants to have an online presence and/or build internal online tools without having to build out DC's everywhere, you would go with cloud.
I could come up with dozens and dozens more. Cloud isn't optimal for everyone, but it is VERY helpful for certain organizations.
>If you're a startup who wants to minimize capex and quickly spin up a reliable, secure infra to host your app on to get it out of the door ASAP, you would go with cloud.
But that's exactly what Etsy isn't. They've been around for years. It's a stable, mature business (at least by the standards of the sort of business they're in). What possible advantage could they gain from switching to this cloud?
Some executive will get to lead a big cloud migration project, which will look great on their résumé when they bounce to their next job a few years later.
Oh, you meant advantages for the company? Never mind.
I chuckled at this comment, but this is actually a very astute observation. I've seen it happen way too many times (and destroy many companies and people's lives) to laugh about it :(.
Access to dozens of services that AWS offers, that you'd want to utilize entirely on their networks ideally. That service list is only going to keep expanding. The cost for Etsy to duplicate and maintain any of those, is high. AWS is also likely to be one of the leading platforms for AI services/tools, which Etsy will no doubt want to make heavy use of in the future; that's another expensive duplication effort if they do it all in-house.
Let's say AWS has 50 services today. Etsy won't have to duplicate any of those by building & maintaining their own versions. It's an extraordinarily time consuming and expensive process to do so. The capabilities AWS can enable will get greater, while the expense to replicate on your own what AWS can do, will also climb. The extrapolated result, is that AWS & the others will perpetually take market share for the next 10+ years from do it yourself approaches. More for the software duplication expense (time & money), than the hardware/bandwidth cost difference.
> If you're a startup who wants to minimize capex and quickly spin up a reliable, secure infra to host your app on to get it out of the door ASAP, you would go with cloud.
Why on earth would a startup waste money like that? Just rent a few servers, most places will have it up in a few minutes, guaranteed, any large provider have automated that long ago. Much, much cheaper than the cloud. It seems, even in this thread, that there are two choices: cloud or colo. Hell no to either. Rent dedicated -- yes.
The only reason one would go with the cloud is if you plan to use their services ie not EC2 but the many others. However, those can have a runaway cost too and it's not clear whether a new startup should take on handcuffs like that -- once you decided you are depending on SQS etc it'll be hard to move to elsewhere.
"If you're a startup who wants to minimize capex and quickly spin up a reliable, secure infra to host your app on to get it out of the door ASAP, you would go with cloud."
I agree, but I don't think that's the market either.
I think the market is in massive big-corps, moving their IT infrastructure to the cloud.
Like regular, non-tech companies.
Small startup bills might be hundreds, thousands, possibly 100's of K but probably not millions. Big corp contracts may be millions - and stable.
At megacorp scales, you're wasting money going to the cloud unless your load profile is highly variable. GCP/AWS profit margins are very high versus bare metal, and if you're megacorp scale you can afford ops and bare metal you can depreciate over several years.
(and before you trot out Netflix, the vast majority of their bits are served out of OpenConnect appliances colocated at eyeball ISPs and peering points)
I recently watched a Mega Corp fire the entire IT Ops department and hire a handful cloud consultants to move all of their current vSphere (thousands of VMs) infrastructure to Azure.
The rationalization was that in the long run it was cheaper to maintain the status quo, and not have to worry about networking, firewall, and replication issues by hand. I'm pretty sure MS also gave them a large discount at that scale.
I don't doubt it. The winds change often, sometimes quickly. People going to move to the cloud, people going to move back to on-prem, as sure as the sun is going to rise.
My hunch is you'll see a lot of move back on-prem during the next downturn when hardware providers need to move a lot of hardware under duress while businesses look at cutting their opex (you have to pay for cloud compute/storage forever, whereas you can run on paid gear for the cost of power, network, cooling, space, and the tech people you need anyway).
To quote Warren Buffet, “You only find out who is swimming naked when the tide goes out."
"and if you're megacorp scale you can afford ops and bare metal you can depreciate over several years."
Don't assume this.
I worked at a 'big tech company' that made a very popular device.
Our IT was a disaster.
We made one of the world's most popular gadgety things - we couldn't make our email work.
Doing things the right way is extremely expensive sometimes, you have to pay zillions for consultants and they can screw it up.
If 'IT' is not a massive cost - often - the consideration is moot - move it to the cloud where things will work, and there is low overhead.
I'll bet $100 that Amazon is earning those margins easily.
Another example: why do companies use Windows instead of Linux which is free? Well, because Linux in a regular corp environment would be a huge nightmare. $50/employee to get something that works? That's dog-food heap. It's hardly a consideration, easy, just buy it. Maybe MS margins are huge. Who cares, because the value they created is monster.
Sometimes 'point and click' IT is worth a lot given all the risk factors etc. etc..
There's also just all sorts of other extra features you may get for free. These are things that may not be worth it for you to spend precious dev time developing, but since cloud providers can make it once and offer it to thousands of customers, it makes much more sense. You can look at the full list of GCP features [0], and it's growing every month.
A critical security issue in public cloud is that users often misunderstand what infrastructure security entails.
I've been in public cloud pitches that sold their security measures as a golden standard that made my managers believe there's no need for a security team anymore.
No need for protocols or proper architecture as x cloud has enough flashy certifications.
I work for a company similar to GrubHub outside the US. 'The Cloud' works great for us. Think, how many people order food at 9AM on a Monday? Therefore, we only have a very small amount of servers running. However, 7PM on a Friday... thousands.
Yes, it's still mega expensive. But allows us to scale when we need it and not have to worry about the hardware, cost, or management of it.
There's an entire other side of building a web server that devs, thankfully, never have to see. OpenCompute [1] aims to standardize what that process looks like. To think that you can run bare metal, and do better than industry standards is naive IMO.
In the end, if you're worrying more about anything than business logic, you're not really delivering value -- you're solving engineering problems. You can sacrifice that control and solving problems for greater abstractions, i.e. serverless, fargate, etc.
Unless you have some extreme security requirements (which most don't) you probably don't have a good reason to run bare metal.
I think you're generalizing too much. The overhead for bare metal only exists for certain use cases and apps.
We run about 3 cages (~4k pieces of hardware total) for about a tenth of the cost of what we've priced out as AWS/GCP. This includes DC staff to swap/provision hardware, leases, capex, and our bandwidth contracts.
Cloud providers have to make money, though. You're doing more with bare metal, yes, but you're not doing much more (especially compared to learning how to use a cloud infrastructure effectively), and the cost may not be worthwhile. Generally, the cost of buying and running one server in a datacenter and staffing the ops team needed to keep the hardware running is cheaper than paying for a 24/7 server of comparable specs in the cloud. And if you're already running in a datacenter, it's extremely not obvious that the cost of moving your workload ever will be net profitable.
Of course, if you don't need comparable specs to an entire physical server (I do a lot of personal stuff on a t2.micro!) or don't need the same capacity 24/7, the math is very different. The cloud is definitely cost-effective for elastic workloads.
> Generally, the cost of buying and running one server in a datacenter and staffing the ops team needed to keep the hardware running is cheaper than paying for a 24/7 server of comparable specs in the cloud
I would disagree here. You can run the metrics for your stack, but it's likely cheaper on AWS.
https://awstcocalculator.com/
If you don't need to compute something but twice a day, or just certain times -- you can pay pennies on the dollar for a serverless function. There's no need for an entire server for things like this.
GCP is an operational expense, whereas bare metal is operational + capital expense. It might be more expensive for cloud, but the accounting looks a little cleaner.
I worked for a company that had bare metal. They were always shipping parts around, lots of movement back and forth between the data center, lots of night deploys for ops team which meant a burden on the HR department. There was a general anxiety about security.
It might be "cheaper" on paper to run bare metal, but it means your managers and executives have to make a lot of dinky little decisions all the time. I'd happily pay more to "push and forget".
I would be shocked if Etsy/Spotify/whoever else wasn't cutting a deal for a significant discount, getting these name brands on GCP is great PR for the platform.
Lifetime costs for much of this technology is more people costs. Also, much of the benefit comes from what's on top of the hosting. (Not having to worry about old versions of software, containers, better security, no issues if the one person who knows what's going on quits, etc)
> I love GCP but honestly have no idea how these big internet properties can make the $ case for “the cloud”, unless they’re also planning to fire 90% of the Ops/SysAdmin team (ala Evernote).
What kind of price differences are you thinking about? How much hardware and how many people are employed to take care of this? Does the tripling of cost take salaries into account?
Having to hire a team that will maintain hardware, deal with scalability and give guaranteed uptime isn't cheap and requires a lot of management. I'm not saying it never makes sense, but offloading all that work on to someone like AWS and Google who have teams doing this for you reliably can be well worth the cost.
most cost cuts come from automation (as you already noticed).
but not just more automation in ops/sysadmin space, but mostly because you can just free resources you don't need.
especially one-off jobs (analytics, etc.) take a huge amount of resources for a very short amount of time and after that time the resources aren't needed.
on clouds you can hope to schedule them preemtivly.
also most cost savings come at night, if you are not active on the whole globe you probably only need half of your resources at night. even if your global the chances are high that at some time you still need way less resources.
etsy is a marketplace on most marketplaces there are times where demand is really high and demand is certainly not so high. (christmas, black firday, regressions, etc..)
also if you self host, high availability can and is a pain.
monitoring your database is a pain, it takes a huge amount of time, which most companies do not have.
p.s. working at a small company and services like RDS/gcloud postgres! (thanks google) google buckets (not sure how they are called their name is not as cool as s3)/S3 are a huge win for us.
Honest question: why not use both? If spinning up cloud computing resources is the most cost-efficient way to do those computing tasks, why not have that be the route for doing one-off computing tasks while your own machines run the workload for recurring stuff? Is there overhead for not having 100% of your computing done on a cloud platform that I'm not aware of or something?
* elastic demand, either in the sense that your global traffic has peaks and troughs, or your global traffic is steady but its global distribution changes such that the balance of where you're deployed would ideally change over time; in either case you can benefit enormously from automatic scaling
* workloads that can take advantage of the spot market (or whatever GCP's equivalent is)
With bare metal you pay for servers you don't use. Cloud is more expensive on the compute side but you can really scale on your usage pattern, also security on the cloud is way better that on bare metal.
These calculations don't make sense unless you really consider TCO over any appreciable amount of time.
It's easy to say you can buy a huge server for a few thousand but the "cloud" provides incredible agility, security, and basically removes operational overhead. When compared to buying, maintaining, and decommissioning servers and software along with the time and personnel costs, it becomes a much closer race. Include global scale, elastic capacity, managed services, and support policies, and it's closer still.
Rarely are companies paying 3x IT spend just to run in the cloud, especially for large companies like Etsy.
We run on bare metal and to move to GCP would triple our hosting cost, not including bandwidth (which is free* for us).
Do people get discounts for doing PR? I’m really curious.