Enterprises Get Access To FBI Search Technology

For the past several years, every time someone at the FBI wanted to search for a name in its Investigative Data Warehouse, they could count on technology from Chiliad working in the background.

Think of it as an uber Google – a search engine capable of pulling information from all of your organisation’s databases.

“The proof is in the pudding,” said Paul McOwen, chief operating officer of Chiliad. “A lot of people can talk about a lot of systems; this one has been in full scale operation for six years [and is] the largest counterterrorism system of its kind at the FBI. When they run 50,000 queries a day, almost all of those queries are going against all billion documents in about 200 different search engine servers, and their average response time is less than three seconds.”

Officials at Chiliad refer to this process as “connecting the dots,” a task sure to challenge organizations as the amount of data in the world continues to expand. Now, after years of focusing on the government sector, Chiliad wants to take their technology into the enterprise space.

“The ‘connecting the dots’ problem is a universal problem in all enterprises whether government or industry-focused,” said Dan Ferranti, CEO of the company. “The bottom line is that every organisation can benefit substantially from being better able to find, fuse, analyze, share and act on information from across the enterprise or across the globe – as if it were all seamlessly contained in a single local real-time application.”

The dot metaphor can be taken quite literally. The company’s platform, dubbed Chiliad Discovery/Alert, works in parallel across distributed repositories of both unstructured and structured data. Rather than moving data across the network to a central indexing system, Chiliad’s technology allows organisations to put a Discovery/Alert node wherever information is managed. Each node is part of a secure peer-to-peer network that allows a query to be executed in parallel across all locations.

“What Chiliad has done is to create a massively parallel processing virtual computer, so when we run a query, the query’s actually delivered to different locations… around the world over multiple networks and that query is fired on the data sets that are behind the repositories in each of the applications,” McOwen said.

The setup allows organizations to avoid problems tied to efficiency and security associated with traditional enterprise search, Chiliad officials said.

“If you search across an Oracle database, with an SQL query, for example, you’re pulling big data through big pipes locally just to get the answer,” he said. “When you search in this parallel virtual computer that Chiliad has designed and built, you’re doing all the work at the endpoint.”

The FBI began using Chiliad’s technology to get around problems involving correlating and accessing data from disparate sources, which is at the heart of what the company does. Enterprises face similar issues as they deal with volumes of unstructured and structured data, and Chiliad has spent the past few months targeting the Fortune 500 with its message.

“We’ve found that the ‘connecting the dots’ problem manifests itself in slightly different ways in specific industries and companies,” Ferranti said. “For example, Chiliad is helping a top pharmaceutical company with investigating and preventing drug counterfeiting. We are also working with multiple insurance companies in order to optimize their claims management process.”

Selling to the enterprise space, however, will mean challenging incumbent vendors adding search into larger platform packages, noted IDC analyst Hadley Reynolds.

“Microsoft, IBM and Oracle have all upgraded their search offering in the past two years, and SAP is also hoping to use this strategy,” he said. “Microsoft in particular, after its acquisition of FAST, has a highly capable enterprise search platform. Since Chiliad’s product is a platform, rather than an application, they will often need to win both a technical evaluation and a vendor viability scan against the larger players in order to be considered.”

Then there is also the issue of shrinking IT budgets as well as the presence of open-source alternatives such as Apache Lucene and Solr, he added.

Still, he referred to Chiliad’s technology as a step towards a larger trend IDC calls “Unified Access.”

“We are going to see a ‘virtualization’ trend in information access that will change the playing field away from traditional enterprise application-based models and toward much more flexible ‘search-like’ intelligence that spans structured and unstructured data and is much more sensitive to a user’s context,” said Reynolds. “Chiliad’s federation architecture is a step in this direction.”