One third of businesses exceeding their cloud budgets by as much as 40%: Survey

More than one-third of businesses have cloud budget overruns of up to 40%, and one in 12 companies exceed this number, according to Pepperdata survey. 

The survey was conducted to better understand how organizations run their big data applications and workloads in the cloud. Companies polled ranged in size from 500 to more than 5000 employees, and spent from USD 500,000 to more than USD 10 M on big data analytics.    

The shift to cloud computing is solidly underway. While the cloud offers both the benefits of a pay-as-you go model and the ability to be elastic on demand, enterprises almost universally see a rise in costs. This is because IT often lacks sufficient visibility into cloud performance and does not have the tools to optimize applications.  

Key findings from the survey include: 

• For 64% of respondents, “cost management and containment” is their biggest concern with running cloud big data technologies and applications. 
• A majority of respondents said the desire to “better optimize current cloud resources” was their highest priority big data cloud initiative.
• In 2020, for one in three respondents, cloud spend was projected to be over budget by between 20% and 40%.  
• One in 12 respondents said their cloud spend was expected to be over budget by more than 40%. 

“This research shows us the importance of visibility into big data workloads. It also highlights the need for automated optimization as a means to control runaway costs,” said Ash Munshi, CEO, Pepperdata. “Significantly more than half the companies surveyed indicate lack of control on cloud spend, making optimization and visibility the keys to getting costs under control.”  

Other findings included: 

• Types of cloud: 47% of respondents are using a private cloud; 21% are in the public cloud; 28% are using a combination of both. 
• Biggest concerns when running big data applications in the cloud: 39% of respondents cited cost management and containment; 33% said increased complexity; 14% said CapEx to OpEx; 13% said a lack of control.
• When asked how budgets were impacted by the move to the cloud, 40% stated that budgets stayed the same and resided with IT; 31% said budget became shared with IT administering; 30% said budgets moved into one or multiple business units. 
• When asked how their enterprise measures application/workload performance based on their cloud instance, 30% said they were using an application performance monitoring solution; 28% said they used cloud provider tools; almost 20% said a homegrown solution; 17% said an observability solution for insights, alerts and automation; five% said they don’t monitor at all. 
• Moving to the cloud often means confusion as to which part of the organization owns support and troubleshooting for big data applications: 44% said they had a shared support model between ITOps and business units/LOB developers; 35% said support stays with ITOps; 22% said the dev organization within the business units owns this.   
• Spend: Asked to estimate what they will spend this year on big data analytics in the cloud, 34% said between USD 500,000 and USD 1 M; 26% said between USD 1 M and USD 2 M; 15% said between USD 2 M and USD 10 M; seven% said more than USD 10 M. 
• Budgets: For 2020, 44% said they’d be on budget; 35% expected to exceed budget by 20 to 40%; eight% of respondents believed they’d exceed budget by more than 40%. 

Cloud optimization delivers big savings. According to Google, even minimal cloud optimization efforts can net a business as much as 10% savings per service within two weeks. Cloud services that are fully optimized and running on extended periods (over six weeks) can save more than 20%. Key findings surrounding cloud optimization were: 

• Top big data cloud initiatives included better optimizing current cloud resources and continuing to migrate workloads into the cloud. Lower-priority initiatives included increasing visibility into app and workload performance; expanding to other public cloud vendors; more use of containers; and finding a reliable replacement for Hadoop. 
• The types of applications and workloads that consumed the most resources, according to respondents were: Hive, with 29% of the vote; Spark, at 27%; MapReduce, at 16%; Tez, at 11%; other applications, 18%.   

To cut the waste out of IT operations processes and achieve true cloud optimization, enterprises need observability and automated tuning. This requires machine learning and a unified analytics stack performance platform. Such a setup equips IT operations teams with the cloud tools they need to keep their infrastructure running optimally while minimizing spend.