Introduction

Early prototype multimedia database management systems used the query-by-example (QBE) paradigm to respond to user queries. Users needed to formulate their queries by providing examples or sketches. Query-by-keyword (QBK) paradigm, on the other hand, has emerged due to the desire to search multimedia content in terms of semantic concepts using keywords or sentences rather than low-level features. This is because it is much easier to formulate some queries by keywords, which is also the way text retrieval systems are working. However, some queries are still easier to formulate by examples or sketches (e.g., trajectory of a moving object). Moreover, there is the so-called "semantic gap" problem, the disparity between the low-level representation and high-level semantics, which makes it very difficult to build multimedia systems capable of supporting keyword-based semantic queries effectively with an acceptable number of semantic concepts. The consequence is the need to support both query paradigms seamlessly in an integrated way.

Another important issue to be considered in today's multimedia management systems is interoperability. This is especially crucial for distributed architectures if the system is to be used by multiple heterogeneous clients. Therefore, MPEG-7 standard as the multimedia content description interface can be employed to address this issue.

The design of a retrieval system is directly affected by the type of queries to be supported. Types of descriptors and the granularity of the representation determine the system's performance in terms of speed and accuracy. As the level of detail in the representation increases more detailed queries can be answered by the system. However, both the database size and system response time increase. Therefore, the system should be designed according to the type of queries to be supported, and representation granularity should be selected accordingly. Below, we give some example video query types that might be attractive for most users, but which also are not supported by the existing systems all together in an MPEG-7 compatible framework.

• Content-based queries by examples. The user may specify an image, an image region or an audio/video segment and the system returns
  video segments similar to the input query.
• Text-based semantic queries. Queries may be specified by a set of keywords corresponding to high-level semantic concepts and relations
  between them.
• Spatio-temporal queries. Queries related to spatial and temporal locations of objects, video segments within the video.
• Composite queries. These queries may contain any combination of other simple queries. The user composes the query
  (hence the name `composite' query) by putting together image/video segments and specifying their properties, and then
  asks the system to retrieve similar ones from the database. This type of queries is especially desirable to formulate very complex queries easily.

We developed BilVideo-7 as a powerful MPEG-7 compatible, distributed video database system to support such multimodal queries in an integrated way. We designed an MPEG-7 profile for video representation which enables detailed queries on videos, and used our MPEG-7 compatible video feature extraction and annotation tool to obtain the MPEG-7 compatible video representations according to this profile. The Visual Query Interface of BilVideo-7 is an easy-to-use and powerful query interface to formulate complex multimodal queries easily, with support for a comprehensive set of MPEG-7 descriptors. Queries are processed on the multi-threaded Query Processing Server with a multimodal query processing and subquery result fusion architecture, which is also suitable for parallelization.