EU Backs IBM’s Multimedia Search Engine

IBM researchers have made a search engine that can identify similar multimedia content such as images and sounds – without using the “tags” relied on by traditional search services such as Google and Microsoft Bing.

SAPIR aims to build search engine which can churn through pixels in images and videos, and the bitstream of audio files, to help users find multimedia content they are looking for, instead of relying on the metadata text tags that are used by traditional search engines such as Google and Microsoft Bing.

The SAPIR project (which stands for Search in Audio-Visual Content Using Peer-to-peer Information Retrieval) is backed by the Chorus research programme funded by the acronym-happy European Union (Chorus stands for “Coordinated approacH to the EurOpean effoRt on aUdio-visual Search engines”, apparently).

IBM and the EU have launched a SAPIR demo Website for public testing [although it appearsto be unavailable at the time of publication].

Traditional search engines Google, Bing and Yahoo rely on text metadata associated with each image to help users find content. If there is no metadata associated with the content, those search engines can’t help users find relevant results, Yosi Mass, a scientist at IBM Research-Haifa and project leader for SAPIR, told eWEEK.

SAPIR indexes the content of each image and clip using descriptors such as text, color, layout, shapes, or sounds, to help users find comparable images. For example, SAPIR scans a digitised photograph or the bit streams in an MP3 sound file, even if they haven’t been tagged or indexed with descriptive information; this is because SAPIR automatically indexes and ranks multimedia content users upload to the Web for easy retrieval.

In a demo, Mass showed how a search on the keyword “dolphin” returned dolphin photos of similar colours and shot angles that users had uploaded to Flickr. After clicking the “similar” link on top of one of the photos, eWEEK saw photos that resembled the dolphin photo in color and shape but did not necessarily include dolphins.

A box labeled “use the image in the search” appeared checked above the SAPIR search, so that when a user types in “dolphin” and clicks the search button, they will see more images of dolphins. This search blends dolphin images with similar images that do not include dolphins. The demo was an exercise in helping users conduct a combination text and image search, something that is not possible with current search engines.

Mass also said SAPIR enables users to introduce latitude and longitude coordinates into searches. Do a search for a topic on SAPIR, click “images” and you’ll see boxes for Lat and Long to the left of a button for Google Maps under the site’s search box.

Enter your geographic area of choice and search for photos on any topic. So, in addition to text and image search, users can further narrow down their topics with location services, which also is crucial for travelers accessing SAPIR via Web-enabled phones.

To wit, this YouTube demo shows how a tourist takes a picture of a statue in Madrid, and searches for comparable pictures in SAPIR. The tourist then did a combined search. By adding “Madrid” to the query, he saw images of the statue he just took a picture of, as well as related pictures of the surrounding plaza he was in.

Eventually, this service could be put to use by cities promoting tourism. “Cities that have a large image collection of monuments in their city can give a very nice service to their tourists,” pairing images with historical text information about the attractions users can access from their Web-enabled phones, Mass explained.

IBM has no concrete plans yet to commercialise SAPIR, but there is a clear value in corralling multimedia content. If IBM can harness some of this unstructured data with the SAPIR technology, it might be most useful for the company’s enterprise search portfolio, as well as for consumer service providers looking to provide their customers with better image and video search.