VMware Project Serengeti To Offer Virtualised Hadoop

VMware has announced a new open-source project, which it is calling Serengeti, which will offer enterprises the ability to quickly deploy, manage and scale Apache Hadoop in virtual and cloud environments.

In addition, VMware announced enhancements to Spring for Apache Hadoop as well as integrations with vSphere and Apache Hadoop to deploy a highly available Hadoop platform.

VMware is working with the Hadoop community to contribute extensions that will make key components “virtualisation-aware” to support elastic scaling and further improve Hadoop performance in virtual environments.

Virtualised Hadoop

“Apache Hadoop has the potential to transform business by allowing enterprises to harness very large amounts of data for competitive advantage,” Jerry Chen, vice president of Cloud and Application Services at VMware, said in a statement. “It represents one dimension of a sweeping change that is taking place in applications, and enterprises are looking for ways to incorporate these new technologies into their portfolios. VMware is working with the Apache Hadoop community to allow enterprise IT to deploy and manage Hadoop easily in their virtual and cloud environments.”

Apache Hadoop is emerging as the de facto standard for big data processing; however, deployment and operational complexity, the need for dedicated hardware, and concerns about security and service-level assurance prevent many enterprises from leveraging the power of Hadoop, VMware said in a press release. By decoupling Hadoop nodes from the underlying physical infrastructure, VMware can bring the benefits of cloud infrastructure – rapid deployment, high-availability, optimal resource utilization, elasticity and secure multi-tenancy – to Hadoop, the company said.

In a blog post about Serengeti, Richard McDougall, CTO for application infrastructure at VMware, said:

“Hadoop gives the ability to store massive amounts of data in a reliable data store, and MapReduce provides a data-parallel programming framework to compute against that data. We have observed that the majority of our customers are using many of the higher-level ecosystem tools, which utilize the power of the underlying data-parallel Hadoop platform through familiar data-access methods – such as Hive for query access, or Pig for script-based data processing.”

Free Download

Available for free download under the Apache 2.0 license, Serengeti is a “one-click” deployment toolkit that allows enterprises to leverage the VMware vSphere platform to deploy a highly available Apache Hadoop cluster in minutes, including common Hadoop components like Apache Pig and Apache Hive. By using Serengeti to run Hadoop on VMware vSphere, enterprises can easily leverage the high-availability, fault-tolerance and live migration capabilities of the world’s most trusted, widely deployed virtualisation platform to enable the availability and manageability of Hadoop clusters.

“Hadoop must become friendly with the technologies and practices of enterprise IT if it is to become a first-class citizen within enterprise IT infrastructure,” Tony Baer, principal analyst at OVUM, said in a statement. “The resource-intensive nature of large big data clusters makes virtualization an important piece that Hadoop must accommodate. VMware’s involvement with the Apache Hadoop project and its new Serengeti Apache project are critical moves that could provide enterprises the flexibility that they will need when it comes to prototyping and deploying Hadoop.”

VMware is working with the leading Apache Hadoop distribution vendors, including Cloudera, Greenplum, Hortonworks, IBM and MapR to support a wide range of distributions.

To further simplify and speed enterprise use of Hadoop, VMware is working with the Hadoop community to contribute changes to the Hadoop Distributed File System (HDFS) and Hadoop MapReduce projects to make them “virtualisation-aware,” so that data and compute jobs can be optimally distributed across a virtual infrastructure. These changes will enable enterprises to achieve a more elastic, secure and high availability Hadoop cluster. The extensions can be found here.

In his post, McDougall, from several perspectives, addressed how and why Hadoop would benefit from virtualisation, saying:

“A full big-data platform typically consists of the Hadoop distributed file system and core Map-reduce, hBase, Pig, Hive, Sqoop and a big-SQL database using traditional SQL or distributed SQL (like Greenplum DB) for more regularly accessed semi-structured data. A good strategy is to architect a common shared platform, on which all of the big-data technologies can reside. By virtualising, all hardware nodes can be common, eliminating the need for special hardware for master services (the NameNode) so that if multiple clusters are deployed, you no longer need to provision and special servers for each of the master services.”

Spring Updates

VMware also announced updates to Spring for Apache Hadoop, an open-source project first launched in February 2012 to make it easy for enterprise developers to build distributed processing solutions with Hadoop. These updates allow Spring developers to easily build enterprise applications that integrate with the HBase database, the Cascading library and Hadoop security.

Spring for Apache Hadoop is free to download and available under the open-source Apache 2.0 license.

Together, these projects and contributions are designed to help accelerate Hadoop adoption and enable enterprises to leverage big data analytics applications, such as Cetas, to obtain real-time, intelligent insight into large quantities of data. VMware acquired Cetas in April, and the Cetas analytics service is available at www.cetas.net.

How well do you know the cloud? Take our quiz and find out!