This version of this document is no longer maintained. For the latest documentation, see http://www.qnx.com/developers/docs.

Multimedia Framework Overview

This chapter provides an overview of the Multimedia Framework, and covers:

Multimedia architecture
QNX-provided components
Extending functionality

Multimedia architecture

The Multimedia library architecture is modular, meaning that it consists of units that can be interchanged with each other to provide only the functionality you need. In addition, because it uses the Addon Interfaces library to define its standard interfaces, you can write new components (such as codecs), and easily incorporate new components written by QNX or third parties as they become available, in some cases without having to recompile your application.

A filter is the basic building block of the Multimedia framework. Multimedia Filters:

are compiled as DLLs or libraries. The DLL versions of the filters are located by default in lib/dll/mmedia (on the target). The library versions are used at compile-time if your application is linking against them statically.
perform a specific task on multimedia data, such as reading a file from disk, parsing, decoding, or writing data to an audio device
are plugins, and therefore implement a standard interface that the Multimedia library understands. The interfaces are defined in the Addon Interface library, aoi.

To process a multimedia file, your application uses only the filters it needs to get the multimedia data, process it, and send it to a target device. We classify filters based on their specific task. A filter can be a:

Reader: Reads data from a source. Examples are audio card readers, data stream readers, and video capture card readers.
Parser: Parses a stream of data into component parts. Examples are MPEG bitstream parsers, WAV format parsers, and AVI format parsers.
Decoder: Decodes compressed or encoded data. Examples are mpeg audio & video decoders, and divx decoders.
Encoder: Encodes raw data into a specific format. Examples are mpeg audio & video encoders.
Writer: Sends data to a destination. Examples are audio card writers, video output writers, and output file writers.

A filter is a generic object. Its functionality is determined by the way it processes data, and the interfaces it implements. A Multimedia library interface defines the set of methods that the library expects filters of a specific type to have. For example, all filters must implement the create and destroy methods defined in the AODeConstructor interface so that the library can create and destroy them, while only filters that read data from a source need to implement the MediaReader interface. The implementation of an interface is declared as a structure of pointers to implemented functions in a defined order.

Each filter can have multiple input and output channels, so if you have a file format that's interleaved with more than one data stream, you can have as many output data channels as the input data stream contains. This is how the MPEG System parser works, for example -- it parses MPEG System data into two streams, video and sound. An MPEG System encoder filter works the opposite way; it has separate input channels for sound and video, and one output channel.

We already provide a number of filters written for the Multimedia framework, but if you need a new one, you have to write only the decoder, parser, or other filter that hasn't already been written. Because the multimedia architecture uses a "plugin" framework, each filter component can easily be integrated (plugged in) with existing filters.

To use the Multimedia library, your application sets up a graph, an abstract object that encapsulates the multimedia filters needed to process a multimedia stream. A graph is stored as a MmGraph_t structure. It's built and destroyed at run time by the player application, and is specific to a particular media stream. Your application adds to the graph all the filters required to process multimedia data from source to destination.

If you know the specific filter required to process the multimedia stream, you can add it to the graph by name. However, a more flexible approach is to use functions that query the library and find the best filter for the data. In this way your application can handle any media type that the library's filters can handle, and use any new filters added to the library without being recompiled.

In the example below, six filters are used to read, parse, decode, and present an MPEG media stream. Each filter has an input channel and at least one output channel, except for special purpose "media reader" and "media writer" filters, which are at the beginning and end of the filter chain.

Multimedia graph example

Multimedia graph example.

In the filters we provide, multimedia streams are synchronized by the audio stream, if present. In the MPEG example above, the writer filters don't communicate directly. Instead, they use the audio writer filter MediaClock interface to the Multimedia library (all filters have hooks to the library) to determine the correct media time. In this way, the video writer continuously checks its media time against the audio writer, and synchronizes its playback accordingly.

QNX components

The Multimedia library includes filters provided by QNX Software Systems that support the multimedia formats listed in the table below. You can write your own filters, or obtain filters from third parties to handle additional formats.

Some filters are available only with the Multimedia Technology Development Kit.

Format	Filter(s)	Mimetype
MPEG-1 (audio)	`mpega_parser.so`, `xing_mpega_decoder.so`	`audio/mp1`, `audio/mp2`, `audio/mp3`, `audio/mpeg`
MPEG-1 (video)	`mpegv_parser.so`, `ff_mpegv_decoder.so`	`video/mpv`
MPEG-1 System (audio and video)	`mpegs_parser.so`, `ff_mpegv_decoder.so`	`video/mpeg`
AU (audio format)	`au_parser.so`	`audio/x-au`, `audio/x-basic`
IFF (audio format)	`iff_parser.so`	`audio/x-iff`
AIFF (audio format)	`aif_parser.so`	`audio/x-aif`
AVI (file format, parser filter only, no codec support)	`avi_parser.so`	`audio/x-avi`
OGG (open source audio format)	`ogg_decoder.so`, `oggi_decoder.so`	`audio/x-ogg`
WAV (audio format, most versions supported)	`wav_parser.so`	`audio/x-wav`
MIDI (audio format)	`midi_parser.so`^*	`audio/midi`

^* The MIDI filter requires a configuration file, midi.cfg, that defines the mapping of MIDI programs to instrument files. See the MIDI Configuration File appendix for more information.

For more detail about each of the filters included with the Multimedia library, see the Multimedia Filter Reference chapter.

The Multimedia library offers these benefits:

In most cases it doesn't require hardware with an FPU to deliver acceptable performance (although hardware with an FPU may deliver better performance).
Its modular design means you include only the media elements you need in your application, which reduces size and complexity.
It allows you to easily write your own codecs to deal with new multimedia data formats, or to rewrite or replace existing codecs.
It can be optimized for memory or CPU performance to suit a wide variety of hardware.
It can handle both frame-based (such as MPEG-1 audio) and streaming (such as MPEG video) data streams.
It's designed with a simple, intuitive API.

Extending functionality

You may want to extend the Multimedia Framework's functionality by writing filters for various media types not handled by the QNX-provided components. The chapter on Extending the Multimedia Framework covers this subject in detail, and shows examples of writing a parser and decoder.

The source for the QNX-provided filters, as well as the source for the Addon Library and Multimedia library, are available in the Multimedia TDK. You require this TDK to write your own filters.