Apache ‘Drill’ Speeds Hadoop Queries

Finding much faster ways to complete Hadoop queries for enterprise users is the aim of “Drill”, the latest open-source project being undertaken by the Apache Software Foundation.

Drill has been established as an Apache Incubator Project, opening its continued development up to software engineers around the world, according to Tomer Shiran, director of product management for Hadoop vendor, MapR Technologies, which is one of the backers of the Apache Drill project.

Open source tools

The Drill project will work to create an open-source version of Google’s Dremel Hadoop tools, which Google uses to speed up its internal use of its Hadoop data analysis tools.

“We’ve spent quite a few months talking to lots of organisations and potential users of Drill and to our customer base as well,” said Shiran, who is a founding member of the Drill project. “We wanted to put this out there as an open-source project, rather than just keep it within MapR for our use alone.”

Drill aids Hadoop users by enabling vastly quicker queries of huge data sets, said Shiran.

“With Drill, you’ll be able to get really fast responses,” he said. Users will be able to get responses within one second, which is a key difference from other tools that are available today, he added.

As it presently works as it was designed, Hadoop does batch processing of large data sets. Drill will improve on that method by doing “interactive analysis” that can find the required answers in the data more quickly, said Shiran. “Interactive analysis is much faster than batch processing.”

The need for tools like Drill has been inspired by always-increasing user requirements, he said. “People have been doing queries in Hadoop, but since it doesn’t return answers to you within a few seconds, it has limitations.”

Ad hoc analysis

Using Drill, users will be able to do ad hoc analysis and get faster responses, whether they are seeking anomalies, data trends or even network intrusions, according to Shiran. “With all of those things, you’re going to have to get a pretty fast response or by the time you do figure it out, it’s going to be old news.”

The nascent Drill open-source project is currently in development and includes a variety of companies and individuals who are working on it right now. “A broad-based effort will be working on this,” said Shiran. “There’s quite a few people actively developing on the project now, so I don’t think it will be a long time before we have an early version released.”

Drill was inspired by Google’s Dremel project, which helps Google perform data analyses on its huge data sets such as analysing crawled web documents, tracking install data for applications on the Android Market, analysing spam, analysing test results on Google’s distributed build system and more, according to Shiran.

By developing Drill as an Apache open-source project, organisers will be able to establish Drill’s own APIs and establish a flexible and robust architecture that will support a broad range of data sources, data formats and query languages, according to the group.

MapR offers two versions of its Hadoop products: MapR M3, which is free; and MapR M5, which is a commercial version of the product with advanced features, including high availability, the ability to make data snapshots and do mirroring of datasets, and 24/7 support.

How well do you know your webcam? Take our quiz.