Using Streams

Updated: October 26, 2022

Streams are off-screen, multi-buffered render targets that are provided by the Screen API.

Streams are used by consumers that are applications rather than Screen itself. In the case of windows, it's most common that the consumer is Screen because the content is intended to be shown on a display. However, the content of streams is intended to be consumed by other applications. These applications could be separate, in the same or different process, from the one that's producing the streams' content. A common case for using streams would be displaying output from a camera (or video) that requires some form of image processing on the camera's output before it's displayed.

Buffers of producer streams are created by calling screen_create_stream_buffers(). These buffers are used for either rendering or posting. Initially, all the buffers that are created, are available for rendering. As the buffers are posted, they become front buffers. The front buffers are once again available as render buffers when all consumers have released them. Render buffers are available for producer streams to safely render into. These buffers are retrieved using the SCREEN_PROPERTY_RENDER_BUFFERS property of the stream. Front buffers have been posted by the stream (i.e., screen_post_stream() was called with these buffers). These buffers are retrieved using the SCREEN_PROPERTY_FRONT_BUFFERS property of the stream. Consumers use the front buffers of producer streams when they are consuming content. We don't recommend rendering into any buffer that's not available in SCREEN_PROPERTY_RENDER_BUFFERS because there's a risk that a buffer that's not in this list, is actively being consumed.

Using streams generally implies a basic set of procedures for both the producer and the consumer. Content can be accessed by multiple applications or processes, so it's important that each of the producers and consumers understand how to provide and access the content. The Screen API provides the capability to perform what each of the producer and consumer are responsible for.

A simple description of a set of producer-consumer functions and interactions is illustrated below to provide a basic understanding of how streams are used to share content. The following steps don't include details and specifics; those are provided in later sections.

  1. Initially, both producer and consumer create their own streams. The producer creates a stream (Stream-p ) with a set of buffers that it will render into. The consumer creates a stream (Stream-c), but doesn't create any buffers because it intends to share those of the producer.

  2. The producer starts rendering into one of its available render buffers. The consumer must call screen_consume_stream_buffers() to associate its stream to the producer's stream. Once the consumer establishes an association with the producer, it can start waiting for content to be posted.

  3. Once the producer is ready to make its content available, it posts the buffer to the stream. The buffer that was used for rendering becomes the front buffer. Once the buffer is posted to the stream, it's available to consumers.

  4. Once a front buffer is available, the consumer must acquire the buffer before it can start processing the buffer's content. By acquiring the buffer, the consumer prevents this buffer from being made available in the producer's list of render buffers (SCREEN_PROPERTY_RENDER_BUFFERS).

  5. When the consumer is finished accessing the buffer from the producer, it needs to release the buffer. By releasing the buffer, it allows the buffer to be updated by the producer.

  6. When the consumer releases the front buffer, the buffer is made available again in the producer's list of render buffers (SCREEN_PROPERTY_RENDER_BUFFERS).

Other important points on streams to note are: