Camera Library Overview

Updated: April 19, 2023

The Camera library provides applications with an interface to interact with cameras connected to your system. The library supports up to 64 cameras but the actual limit depends on your hardware platform's limitations and resources.

Each application can support up to 16 cameras. If you had 64 cameras on your system and wanted to use all 64, you could do so only if you had four applications running and each application used 16 cameras.

The Camera library simplifies the process of interacting with connected cameras. You can use this library to build applications that:

Camera support components



Figure 1. Camera support components
Camera support is provided by the following components:
Sensor service
The Sensor service is a process that interacts directly with the camera hardware. The service runs initially as root, gets the required privileges, and then runs as non-root (UID/GID 521).

The camera hardware is configured in a separate configuration file. This file allows you to change camera settings and determines the identifiers to use when you work with the cameras using the Camera API.

The Sensor service does the following:
  • processes requests from applications that use the Camera library API
  • tracks the permissions and access of applications to cameras in the system
  • ensures that the client applications have the necessary permissions to access a camera
  • manages the power to a camera based on the connections to it
  • manages the camera configuration settings

The Sensor service also handles the main interactions with the framework, including interactions with the Screen service, and managing a viewfinder window and Stream object of the image buffers from a camera.

Sensor configuration file
A text-based file used to configure the cameras on the target board. You can specify the name and additional metadata to describe the position, direction, and settings of a camera. Cameras must be configured based on the connection type.

In addition to the connection types, you can configure a file camera, which is a virtual camera that plays a video file to simulate a video stream, as if it came from a live camera. You'll find that file cameras are very useful for doing developer testing.

For more information about the configuration file, see “Sensor configuration file” in the Sensor chapter of the Sensor Framework Services Guide.

Camera library
A C-based API to interface with the connected cameras. The library provides high-level functions to work with cameras, including starting and closing cameras, changing focus, zooming, etc.

The Camera library allows you to encode and decode H.264-formatted files. It also supports uncompressed video, both UCV and MOV files. The libraries required are included with your Sensor Framework installation and found in $QNX_TARGET/ARCH/libs. For example, for x86_64 targets, it would be $QNX_TARGET/x86_64/libs.

Capabilities of the Camera library

The Camera library lets your applications:
  • connect and disconnect from a camera
  • start and stop the viewfinder, which is our abstraction of an image buffer or a video stream
  • use camera features, such as zoom, autofocus, auto white balance, and auto exposure
  • adjust focus, exposure, and white balance algorithms by defining regions of interest, or applying manual controls
  • access the stream of image buffers (framebuffers)
  • encode video and save recordings to files
Note: The Camera library doesn't automatically play audible sounds when it starts to record video. Applications must supply their own sound when a video recording starts or ends. While you can choose to not have an audible sound heard when you start or stop a recording, you are responsible to ensure that the application adheres to the local laws of the regions in which you wish to distribute it. For example, it's illegal to mute or modify sound of a camera application in Japan and Korea, so you must comply with those laws if you distribute your software in those countries.

The Camera library lets application developers implement vision capabilities for their system. In particular, the library provides the camera input, video capture, video playback, and support for hardware-accelerated vision algorithms on supported target boards.

Interaction with other QNX components

To use the Camera library, you should have a basic understanding of how to work with the Screen Graphics Subsystem (Screen). Depending on what your application does, you may need to have knowledge of the Screen concepts described in the sections that follow.

Camera library header files

To use the Camera library, these are the header files that you can use:
camera_api.h
This is the main interface to the Camera library. You must always include this header file to use the Camera library.
camera_encoder.h
This interface is used to get and set video-encoding properties.
camera_h264avc.h
This interface defines the encoding properties that are used with the camera_encoder.h interface.
camera_3a.h
This interface is used to work with various manual settings on the camera that include ISO, shutterspeeds, white balance, and aperture.
external_camera_api.h
This interface allows you to write your own camera driver.

Working with cameras

The Camera library supports many different cameras, such as ISPs, USB, CMOS sensors, and System on a Chip (SoC) systems. Because of this, the Camera library supports decoding and encoding on different hardware platforms and hardware platform variants. When the Sensor service runs, it knows which cameras are available based on what's specified in the sensor configuration file.

After you start the Sensor service with a valid sensor configuration file, you can use the Camera library to:
  • query what cameras are available, as well as metadata (if available in the configuration file) regarding each camera
  • connect to a camera
  • configure a camera; that includes framerates, resolution, etc.
  • perform operations with a camera, such as showing live video, recording, and encoding video

Working with image buffers

The library allows you to access the image stream (or video stream), which we refer to as image buffers in the context of the Camera library. With the image buffers, you can build applications that:
  • Process the image buffers using third-party algorithms to get information about objects in these buffers.
  • Access camera frames. You can do this using one of two mechanisms, either callbacks or events. For more information, see “Using callback mode” and “Using event mode.”
  • Access the image buffers using streams, which allows you to process these buffers using the GPU (if available on your target). This is done using streams from the Screen Graphics Subsystem. For more details, see “Using stream mode.”
  • Display a live preview (unprocessed image buffers). In the context of the Camera library, we refer to this as a viewfinder. For more information, see “Viewfinder Mode.”
  • Encode the image buffers to H.264-formatted frames that can be sent to another component on a network bus or saved to a file. The Camera library also allows you decode an H.264-formatted MP4 file.
  • Record and play uncompressed video as both UCV and MOV files. MOV files are a standard format that can be played on a PC, however they support only RGB 32-bit and YCrYCb 4:2:2 packed formats. If the camera you use doesn't support one of those formats, you must use UCV. UCV is a proprietary format that can't be played on your PC, but you should be able to play any other video format supported by the Camera library.