Introduction
NVIDIA DeepStream SDK is a powerful framework for building high-performance, AI-powered video analytics applications. While DeepStream offers a rich set of pre-built GStreamer plugins for common tasks like decoding, inferencing, and tracking, real-world AI solutions often demand unique processing steps or custom model integrations. This tutorial empowers developers to extend DeepStream's capabilities by creating custom GStreamer plugins, unlocking advanced AI inference and tailored video processing workflows.
In this comprehensive guide, you will learn the fundamental concepts of GStreamer plugin development, understand how to integrate these plugins seamlessly into DeepStream pipelines, and gain the practical skills to implement your own custom logic. We'll cover everything from setting up your development environment to coding, building, and debugging your custom GStreamer element. By the end, you'll be equipped to tackle complex AI video analytics challenges with DeepStream.
Prerequisites: A Linux environment (Ubuntu 18.04/20.04 recommended), NVIDIA DeepStream SDK installed, basic familiarity with GStreamer concepts, and proficiency in C/C++ programming. Time Estimate: This tutorial is comprehensive and involves coding; expect to spend approximately 3-5 hours to follow along and complete the examples.
What is NVIDIA DeepStream SDK?
NVIDIA DeepStream SDK is a complete streaming analytics toolkit for AI-based video and image understanding. Built on top of the GStreamer framework, it provides a highly optimized and accelerated pipeline for processing video streams using NVIDIA GPUs. DeepStream simplifies the development of complex applications by offering a collection of GStreamer plugins that handle various stages of a video analytics pipeline, from input and decoding to neural network inference, object tracking, and output visualization.
At its core, DeepStream leverages the power of NVIDIA GPUs and TensorRT for high-performance inference, enabling developers to deploy AI models efficiently on edge devices or in the cloud. It supports various input sources like RTSP streams, local files, and USB cameras, and provides mechanisms for metadata generation and handling, which is crucial for subsequent processing steps. The SDK's modular nature, inherited from GStreamer, allows developers to construct pipelines by chaining together different elements, making it flexible and scalable for diverse use cases.
DeepStream's primary advantage lies in its ability to process multiple video streams simultaneously with high throughput and low latency, making it ideal for applications such as smart cities, industrial inspection, retail analytics, and intelligent transportation. Its architecture is designed to maximize GPU utilization, minimizing CPU overhead and ensuring that AI inference is performed as efficiently as possible. This robust foundation is what we aim to extend with custom GStreamer plugins, tailoring its capabilities precisely to unique project requirements.
Why Customize DeepStream Inference?
While DeepStream provides powerful inference capabilities through its `nvinfer` (for primary inference) and `nvinferserver` (for secondary inference) plugins, there are often scenarios where out-of-the-box solutions aren't sufficient. Customizing DeepStream inference, or the post-processing of inference results, becomes essential when you need to implement unique business logic, integrate specialized AI models, or handle complex data transformations that are not natively supported by the SDK's standard components.
One common reason for customization is the need for advanced post-processing of detection or classification results. For instance, you might require custom filtering rules based on object attributes, spatio-temporal analysis across frames, or combining inference outputs from multiple models in a non-standard way. DeepStream's default plugins provide basic filtering and tracking, but highly specific application logic, such as counting objects entering specific zones or triggering alerts based on complex event patterns, often necessitates a custom plugin. This allows you to inject your unique algorithms directly into the DeepStream pipeline, leveraging its performance benefits.
Furthermore, if you're working with a highly specialized neural network architecture or a model trained on a peculiar dataset that produces non-standard output formats, a custom GStreamer plugin can act as an adapter. It can parse these unique outputs, convert them into DeepStream's standard metadata format (NvDsBatchMeta, NvDsObjectMeta, etc.), or perform additional computations before passing the data down the pipeline. This flexibility ensures that DeepStream remains adaptable to the cutting edge of AI research and proprietary model deployments, making custom plugin development a crucial skill for advanced AI video analytics DeepStream applications.
Step-by-Step Guide: Building a Custom GStreamer Plugin
This section will guide you through the process of creating a simple custom GStreamer plugin for DeepStream. We'll build a plugin that can read DeepStream metadata and perform a basic operation, demonstrating the core concepts. For this tutorial, we will create a plugin called gst-nvds-example that logs the number of detected objects per frame.
1. Setting Up Your Development Environment
Before you begin coding, ensure your system is properly configured for GStreamer and DeepStream development. This involves verifying DeepStream installation and installing necessary development libraries.
-
Verify DeepStream SDK Installation:
Ensure that NVIDIA DeepStream SDK is correctly installed on your system. You can check by trying to run one of the sample applications or by verifying the presence of DeepStream libraries.
$ deepstream-app --version # Or check gstreamer plugins $ gst-inspect-1.0 nvinfer[IMAGE: DeepStream version check output]
If DeepStream is not installed, please follow the official NVIDIA documentation for installation relevant to your platform (Jetson, dGPU, or Docker).
-
Install GStreamer Development Packages:
You'll need GStreamer development headers and libraries, along with the Autotools build system components. For Ubuntu, these can be installed via:
$ sudo apt-get update $ sudo apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev \ libgstreamer-plugins-good1.0-dev autoconf automake libtool build-essential \ git pkg-config flex bison[IMAGE: Terminal showing GStreamer dev package installation]
These packages provide the necessary tools and headers for compiling GStreamer plugins.
pkg-configis crucial for finding library paths during compilation. -
DeepStream Development Headers:
Ensure that the DeepStream development headers are accessible. These are typically installed with the SDK in `/opt/nvidia/deepstream/deepstream-X.Y/sources/includes/` and `/opt/nvidia/deepstream/deepstream-X.Y/include/`.
Pro Tip: Using a DeepStream Docker container can simplify environment setup significantly, as it comes pre-configured with most necessary dependencies.
2. Understanding GStreamer Plugin Basics
Developing GStreamer plugins involves understanding its object model and how elements interact within a pipeline. Every GStreamer plugin is essentially a shared library containing one or more elements. These elements are the building blocks of GStreamer pipelines, performing specific tasks like decoding, filtering, or rendering.
At its core, a GStreamer element is a GstElement, which is an object derived from GObject (GLib's object system). Plugins often inherit from base classes like GstBaseTransform for in-place data modification, GstBaseSrc for source elements, or GstBaseSink for sink elements. Our example will use GstBaseTransform as it's suitable for processing data already in the pipeline, such as DeepStream metadata.
Key concepts for plugin development include:
- Elements: The functional units of a pipeline. Each element has a unique name and defines its capabilities.
- Pads: Connection points on elements. Source pads output data, and sink pads accept data. Pads have "capabilities" (
GstCaps) which describe the type of media data they can handle (e.g., video/x-raw, width=1920, height=1080). - Capabilities (GstCaps): Describe the media formats an element can process or produce. During pipeline construction, GStreamer negotiates capabilities between connected pads to ensure compatibility.
3. Designing Your Custom Plugin
For our example, we'll design a plugin that acts as a simple DeepStream metadata logger. It will receive video frames along with their associated DeepStream metadata, count the number of objects detected in each frame, and print this count to the console. This plugin will be placed after an inference element (like nvinfer) in a DeepStream pipeline.
Plugin Name: nvds-example
Purpose: Log the number of detected objects per frame from DeepStream metadata.
Input Capabilities: This plugin will expect DeepStream-compatible video buffers. Specifically, it should accept buffers with the video/x-raw format, indicating raw video frames, and crucially, it must be able to access the attached NvDsBatchMeta and NvDsFrameMeta metadata.
Since our plugin only reads metadata and doesn't modify the video frame data itself, it will act as an "in-place" transformer. This means it will receive a buffer, process its metadata, and then pass the same buffer downstream without creating a new one. This is efficient and a common pattern for metadata-only operations within DeepStream.
GStreamer for AI video processing: GStreamer's extensible architecture, combined with its robust media handling capabilities, makes it an excellent choice for AI video processing. Custom plugins allow developers to inject AI-specific logic, such as pre-processing inputs for neural networks, integrating custom inference engines, or performing sophisticated post-analysis on AI model outputs, all within a high-performance, stream-based framework.
4. Implementing the Plugin Code
Now, let's dive into the code. We'll create a single C file (e.g., gstnvds_example.c) for our plugin. A typical GStreamer plugin involves several boilerplate structures and functions.
4.1. Boilerplate and Plugin Registration
Every GStreamer plugin starts with common boilerplate for GObject and GStreamer element registration.
#include <gst/gst.h>
#include <gst/base/gstbasetransform.h>
#include <stdio.h>
// DeepStream specific headers
#include <nvds_version.h>
#include <nvds_logger.h>
#include <nvds_utils.h>
#include <nvds_meta.h>
// Define the element's class and instance structures
typedef struct _GstNvDsExample GstNvDsExample;
typedef struct _GstNvDsExampleClass GstNvDsExampleClass;
struct _GstNvDsExample {
GstBaseTransform parent;
gint frame_num;
};
struct _GstNvDsExampleClass {
GstBaseTransformClass parent_class;
};
// Element registration macro
#define GST_TYPE_NVDS_EXAMPLE (gst_nvds_example_get_type())
#define GST_NVDS_EXAMPLE(obj) (G_TYPE_CHECK_INSTANCE_CAST((obj),GST_TYPE_NVDS_EXAMPLE,GstNvDsExample))
#define GST_NVDS_EXAMPLE_CLASS(klass) (G_TYPE_CHECK_CLASS_CAST((klass),GST_TYPE_NVDS_EXAMPLE,GstNvDsExampleClass))
#define GST_IS_NVDS_EXAMPLE(obj) (G_TYPE_CHECK_INSTANCE_TYPE((obj),GST_TYPE_NVDS_EXAMPLE))
#define GST_IS_NVDS_EXAMPLE_CLASS(klass) (G_TYPE_CHECK_CLASS_TYPE((klass),GST_TYPE_NVDS_EXAMPLE))
#define GST_NVDS_EXAMPLE_CAST(obj) ((GstNvDsExample *)(obj))
// Forward declarations
G_DEFINE_TYPE (GstNvDsExample, gst_nvds_example, GST_TYPE_BASE_TRANSFORM);
static void gst_nvds_example_class_init (GstNvDsExampleClass * klass);
static void gst_nvds_example_init (GstNvDsExample * filter);
static GstFlowReturn gst_nvds_example_transform_ip (GstBaseTransform * trans, GstBuffer * buf);
static gboolean gst_nvds_example_set_caps (GstBaseTransform * trans, GstCaps * incaps, GstCaps * outcaps);
// Capabilities declaration
static GstStaticPadTemplate sink_factory = GST_STATIC_PAD_TEMPLATE ("sink",
GST_PAD_SINK,
GST_PAD_ALWAYS,
GST_STATIC_CAPS ("video/x-raw(memory:NVMM), "
"format = (string) { NV12, P010_10LE }, "
"width = (int) [ 1, MAX ], "
"height = (int) [ 1, MAX ]")
);
static GstStaticPadTemplate src_factory = GST_STATIC_PAD_TEMPLATE ("src",
GST_PAD_SRC,
GST_PAD_ALWAYS,
GST_STATIC_CAPS ("video/x-raw(memory:NVMM), "
"format = (string) { NV12, P010_10LE }, "
"width = (int) [ 1, MAX ], "
"height = (int) [ 1, MAX ]")
);
// Plugin initialization
static gboolean
plugin_init (GstPlugin * plugin)
{
return gst_element_register (plugin, "nvds-example", GST_RANK_NONE, GST_TYPE_NVDS_EXAMPLE);
}
// Plugin description
GST_PLUGIN_DEFINE (
GST_VERSION_MAJOR,
GST_VERSION_MINOR,
nvds_example,
"NVIDIA DeepStream Example Plugin",
plugin_init,
NVDS_VERSION_MAJOR "." NVDS_VERSION_MINOR,
"LGPL",
"NVIDIA",
"http://nvidia.com/"
)[IMAGE: Code snippet for plugin boilerplate and registration]
This code defines our element's types, registers it with GStreamer, and declares its input/output capabilities. Notice the use of video/x-raw(memory:NVMM), which is crucial for DeepStream elements to process frames directly on GPU memory without costly host-to-device transfers.
4.2. Class and Instance Initialization
The class_init and init functions are called when the plugin is loaded and when an instance of the element is created, respectively.
static void
gst_nvds_example_class_init (GstNvDsExampleClass * klass)
{
GstBaseTransformClass *base_transform_class = GST_BASE_TRANSFORM_CLASS (klass);
GST_DEBUG_CATEGORY_INIT (gst_nvds_example_debug, "nvds-example", 0, "NVIDIA DeepStream Example Plugin");
gst_element_class_add_pad_template (GST_ELEMENT_CLASS (klass),
gst_static_pad_template_get (&sink_factory));
gst_element_class_add_pad_template (GST_ELEMENT_CLASS (klass),
gst_static_pad_template_get (&src_factory));
gst_element_class_set_static_metadata (GST_ELEMENT_CLASS (klass),
"NVIDIA DeepStream Example Plugin", "Transform",
"Example plugin for DeepStream metadata processing",
"Your Name <your.email@example.com>");
base_transform_class->transform_ip = GST_DEBUG_FUNCPTR (gst_nvds_example_transform_ip);
base_transform_class->set_caps = GST_DEBUG_FUNCPTR (gst_nvds_example_set_caps);
}
static void
gst_nvds_example_init (GstNvDsExample * filter)
{
filter->frame_num = 0;
}[IMAGE: Code snippet for class and instance initialization]
In class_init, we set up pad templates, metadata for gst-inspect-1.0, and most importantly, override the transform_ip function (our core processing logic) and set_caps. The init function is where we initialize instance-specific data, like our frame counter.
4.3. Capability Negotiation (set_caps)
The set_caps function is called during pipeline negotiation to confirm that the plugin can handle the incoming media format.
static gboolean
gst_nvds_example_set_caps (GstBaseTransform * trans, GstCaps * incaps, GstCaps * outcaps)
{
GST_DEBUG_OBJECT (trans, "Set caps to %" GST_PTR_FORMAT, incaps);
// For a simple passthrough transform, we just accept the caps if they match our templates.
// In more complex scenarios, you might analyze caps properties here.
return TRUE;
}[IMAGE: Code snippet for set_caps function]
For a simple GstBaseTransform that doesn't change the buffer format, set_caps can often just return TRUE if the incoming caps are compatible with the pads' static templates.
4.4. Core Processing Logic (transform_ip)
This is where the magic happens. The transform_ip function is called for each buffer that flows through the element. Here, we access DeepStream metadata.
static GstFlowReturn
gst_nvds_example_transform_ip (GstBaseTransform * trans, GstBuffer * buf)
{
GstNvDsExample *filter = GST_NVDS_EXAMPLE (trans);
NvDsBatchMeta *batch_meta = NULL;
NvDsFrameMeta *frame_meta = NULL;
guint num_frames_in_batch = 0;
guint num_objects_in_frame = 0;
gpointer state_ptr = NULL;
filter->frame_num++;
// Retrieve batch metadata from the GstBuffer
batch_meta = gst_buffer_get_nvds_batch_meta (buf);
if (!batch_meta) {
GST_WARNING_OBJECT (filter, "NvDsBatchMeta not found on buffer %p", buf);
return GST_FLOW_OK; // Continue processing even if no metadata
}
// Iterate through all frames in the batch
for (frame_meta = nvds_get_nth_frame_meta (batch_meta, 0);
frame_meta != NULL;
frame_meta = nvds_get_next_frame_meta (frame_meta)) {
num_frames_in_batch++;
num_objects_in_frame = 0;
// Iterate through all objects in the current frame
for (NvDsObjectMeta *object_meta = nvds_get_nth_object_meta (frame_meta->obj_meta_list, 0);
object_meta != NULL;
object_meta = nvds_get_next_object_meta (object_meta)) {
num_objects_in_frame++;
}
// Log the results
g_print ("Frame %d (Stream ID: %d): Detected %d objects.\n",
frame_meta->frame_num, frame_meta->pad_index, num_objects_in_frame);
// Example of accessing other metadata (e.g., source info)
NvDsSourceMeta *source_meta = nvds_get_source_meta_from_frame_meta(frame_meta);
if (source_meta) {
// g_print("Source ID: %d, URI: %s\n", source_meta->source_id, source_meta->uri);
}
}
// You can also iterate through other metadata types if needed
// NvDsUserMeta *user_meta = NULL;
// for (user_meta = batch_meta->user_meta_list; user_meta != NULL; user_meta = user_meta->next) {
// // Process user metadata
// }
return GST_FLOW_OK;
}[IMAGE: Code snippet for transform_ip function accessing DeepStream metadata]
In this function:
- We increment a frame counter.
- We use
gst_buffer_get_nvds_batch_meta(buf)to retrieve the DeepStream batch metadata attached to the GStreamer buffer. - We then iterate through the
NvDsFrameMetalist within the batch, and for each frame, iterate through itsNvDsObjectMetalist to count objects. - Finally, we print the count.
5. Building and Installing Your Plugin
GStreamer plugins are typically built using Autotools (configure.ac, Makefile.am) or Meson. For simplicity, we'll use a basic Makefile here, but for production, Autotools or Meson is recommended.
5.1. Basic Makefile
Create a file named Makefile in the same directory as gstnvds_example.c:
PLUGIN_NAME := nvds_example
PLUGIN_SOURCES := gstnvds_example.c
PLUGIN_LIB := libgst$(PLUGIN_NAME).so
# GStreamer 1.0 flags
GST_CFLAGS := $(shell pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-plugins-base-1.0)
GST_LIBS := $(shell pkg-config --libs gstreamer-1.0 gstreamer-base-1.0 gstreamer-plugins-base-1.0)
# DeepStream specific flags
# Adjust DEEPSTREAM_PATH if your DeepStream installation is in a different location
DEEPSTREAM_PATH := /opt/nvidia/deepstream/deepstream-6.3
NVDS_CFLAGS := -I$(DEEPSTREAM_PATH)/sources/includes -I$(DEEPSTREAM_PATH)/include
NVDS_LIBS := -L$(DEEPSTREAM_PATH)/lib -lnvdsgst_meta -lnvds_meta -lnvds_utils
# Compiler and linker flags
CFLAGS := $(GST_CFLAGS) $(NVDS_CFLAGS) -fPIC -Wall -Werror
LDFLAGS := $(GST_LIBS) $(NVDS_LIBS) -shared -Wl,--no-undefined
all: $(PLUGIN_LIB)
$(PLUGIN_LIB): $(PLUGIN_SOURCES)
$(CC) $(CFLAGS) $(PLUGIN_SOURCES) -o $(PLUGIN_LIB) $(LDFLAGS)
clean:
rm -f $(PLUGIN_LIB)
install: $(PLUGIN_LIB)
# Install to user's GStreamer plugin path
mkdir -p ~/.config/gstreamer-1.0/plugins
cp $(PLUGIN_LIB) ~/.config/gstreamer-1.0/plugins/
# You might need to run: gst-inspect-1.0 --atleast-version 1.0 &> /dev/null
# Or simply restart your terminal/application for GStreamer to find new plugins.
.PHONY: all clean install[IMAGE: Code snippet for the Makefile]
Important: Adjust DEEPSTREAM_PATH to match your DeepStream installation path.
5.2. Compile and Install
Open your terminal in the directory containing your .c file and Makefile, then run:
$ make
$ make install[IMAGE: Terminal showing make and make install commands]
The make command compiles your C code into a shared library (libgstnvds_example.so). The make install command copies this library to your user-specific GStreamer plugin directory (~/.config/gstreamer-1.0/plugins/), making it discoverable by GStreamer applications.
5.3. Verify Installation
After installation, use gst-inspect-1.0 to verify that GStreamer recognizes your new plugin:
$ gst-inspect-1.0 nvds-example[IMAGE: Terminal showing gst-inspect-1.0 output for nvds-example]
You should see detailed information about your plugin, including its pads, capabilities, and descriptions. If you see "No such element or plugin 'nvds-example'", double-check your installation path and that the plugin name in gst_element_register matches.
6. Integrating with a DeepStream Pipeline
Now that your plugin is built and installed, let's integrate it into a DeepStream pipeline. We'll use a simple gst-launch-1.0 command for demonstration.
# Example DeepStream pipeline with your custom plugin
# This pipeline assumes a test video file and a pre-trained model for nvinfer.
# Adjust paths as necessary for your DeepStream installation.
# For DeepStream 6.x
# Input source (e.g., sample H.264 video)
SOURCE_URI="file:///opt/nvidia/deepstream/deepstream-6.3/samples/streams/sample_720p.h264"
# Primary GIE model (e.g., PeopleNet)
PGIE_CONFIG_FILE="/opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/config_infer_primary.txt"
gst-launch-1.0 filesrc location=$SOURCE_URI ! \
qtdemux ! h264parse ! nvv4l2decoder ! \
m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! \
nvinfer config-file-path=$PGIE_CONFIG_FILE ! \
nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream-6.3/lib/libnvds_nvmultiobjecttracker.so \
ll-config-file=/opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/config_tracker_IOU.yml ! \
nvds-example ! \
nvdsosd ! nveglglessink -e[IMAGE: Diagram of DeepStream pipeline with custom plugin highlighted]
In this pipeline:
filesrcandqtdemux/h264parse/nvv4l2decoderhandle video input and decoding.nvstreammuxbatches frames from multiple sources (here, just one).nvinferperforms primary object detection (e.g., using a PeopleNet model). This element attachesNvDsBatchMetawithNvDsObjectMetato