Scripted Streaming - Teknique´s camera modules
By Stefan Tauschek, Macnica GmbH
Translation: Andrew Benett, Macnica UK
Now that camera technology has completely transitioned to digital and capable broadband infrastructures are available, it has spurned the growth of IP cameras into numerous markets, such as the well-known action cams on motorcycles and sport helmets as well as surveillance and security with the likes of dashcams and public and private monitoring areas.
While earlier development of cameras using analogue tubes or CCDs was reserved for selected specialists with a lot of relevant know-how, the situation has changed fundamentally since then. Software skills are more important today, because almost all implementations are based on a Linux Kernel with stacked middleware for exporting the hardware functions and an application layer running the actual camera application. Although knowledge of optics and especially image sensor technology are still necessary "skills" for a camera designer, the knowledge requirement expands to include interfaces, network connectivity and more recently AI (Artificial Intelligence) with its enormous potential in industrial video analytics becomes more prevalent.
Current camera designs use system-on-chips (SoCs) with integrated controllers, encoders and DSPs which can greatly simplify product development. The US based manufacturer Ambarella is one to mention, as their SoCs are still world-class in terms of quality and power efficiency. Nevertheless, even with these highly integrated components, it is often a long R&D cycle before a camera is ready for sale and as such these projects world normally require high volume production.
For smaller quantities, the New Zealand company Teknique now offers modular cameras based on the Ambarella SoCs enabling successful product placement with a short "time-to-market". It is not only the hardware which is already functional "out-of-the-box" but more important, a smart software strategy with the possibility of application development at only the complexity of bash scripts.
The S5L is a 5th Generation IP camera SoC fabricated in 14nm LPCMOS with powerful image and video processing capabilities with minimal power consumption. For example, the S5Lm variant can be used with slightly reduced clock rates of the video pipeline and the ARM controller for battery-powered cameras.
Basic functional blocks of an IP camera SoC are the Image DSP, also called ISP or sensor pipeline, the video DSP for encoding, a powerful LPDDR memory controller and an ARM processor for central control of all processes. Since all compute-intensive processes for image processing are outsourced to signal and vector processors, the CPU is hardly stressed even in load operation with, for example, 4kp30 H.265 encoding and still offers sufficient power reserves for customer-specific applications.
The Image-DSP forms the interface to the image sensors and its performance is crucial not only for the image quality but also for the compression quality due to its sophisticated 3D-MCTF noise reduction. Any image noise significantly lowers the compression quality, as it inflates the necessary data rate without transmitting more image details.
But the sensor pipeline even does more, such as the conversion of sensor raw data into YUV or RGB values, the dynamics improvement by HDR or WDR methods, the geometric correction of wide-angle recordings (de-warping), electronic image stabilization, defogging and optimal setting of image parameters like brightness / contrast / color balance.
The Video DSP encodes the images stored in the main memory into standard compliant JPEG, H.264 or HEVC video streams. For maximum compression efficiency, it features advanced motion detection, adjustable GOP structures and thus achieves extremely low data rates: Given the SmartAVC, an Ambarella modified H.264 encoder, a 1080p30 video without any moving image content can be transmitted at 100 kbps. With motion video content, data rate goes up to about 400-800 kbp/s with acceptable quality. Using SmartHEVC based on H.265 encoding standard data rates may be reduced by another 50%. This compression capability not only enables the deployment of high-resolution IP cameras over narrow-band infrastructures (such as radio links), but also saves considerable storage space when recording video material.
The GStreamer project was founded back in 1999 and has been further developed with the support of RidgeRun since January 2001. RidgeRun, a company for embedded Linux implementations, especially focused on GStreamer for use in mobile phone media applications. Since then, the project has increasingly found support in the open source community and is now available in release 1.16. GStreamer is free open source software under the terms of the GNU Lesser General Public License (LGPL) and is available on gstreamer.freedesktop.org.
GStreamer processes media streams through process elements in the form of plugins in a pipeline. Elements are connected via pads, means a connection is made from the src pad (source) of an element to the sink pad (sink) of the successor element. The data communication between sink and source pads is exchanged and negotiated via capabilities.
A simple example shows figure 5: A source videotestsrc (test image) is connected to a sink autovideosink (display). In order to realize this pipeline, the gstreamer environment has the command line tool 'gst-launch-xx'. This allows the pipeline to be set up as follows:
gst-launch-1.0 videotestsrc ! autovideosink
The command "gst-launch-1.0" uses exclamation marks to connect two elements. If there are more than 2 elements, the command line contains further elements concatenated with "!" Characters.
teknique has now extended the gstreamer environment with plugins and applications based on the embedded Ambarella SDK. This can be used to initiate a video stream via the local network, for example with the following command:
gst-launch-1.0 oclea_video_src ! video/x-h264, width=1920, height=1080, stream-format=byte-stream ! rtph264pay ! udpsink host=192.168.100.123
The video source "oclea_video_src" with configuration 1920x1080 px (FullHD) and byte-stream feeds the H.264 encoder, from there exports a RTP H.264 payload and streams over UDP on the host with local IP address 192.168.100.123. On the destination host, the stream can then be displayed e.g. with the VLC player, but the GStreamer environment also provides a way to display:
gst-launch-1.0 rtpbin name=rtpbin udpsrc caps=’application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264’ port=5004 ! rtpbin.recv_rtp_sink_0 rtpbin. ! rtph264depay ! decodebin ! videoconvert ! autovideosink
GStreamer relies on a plugin architecture and its functionality is mostly implemented as a "shared library". Thus, the framework is largely platform-agnostic, i.e. can be used in different operating system and media environments, so it is ideal for use in an Ambarella-based module camera.
Teknique significantly helps with accelerating Ambarella-based cameras application development. Complex tasks of image processing such as setting up an image pipeline or an H.264 codec is now available as script commands, and entire application chains aka pipelines can be described in a single line.
Although not all features of Ambarella’s SDK are available in the current revision r 0.9.2 of the Oclea SDK, Teknique announces full implementation for the upcoming r 1.0 release.
Scripting with "gst-launch-1.0" is a fast method of evaluating camera features but less the tool of choice for developing an application. For this purpose, Teknique provides a docker image that provides the necessary build environment with CMAKE in addition to the libraries. Installed on a development host with Linux OS (Ubuntu is recommended), the application development coded in "C" is possible.
tekniques modular cameras can be combined from a morphological building set of individual components - in the S5L μSoM universe, there are a number of image sensor modules with different optics for plugging onto the S5L module including carrier plate and finally base modules with different interfaces such as wired or wireless LAN.
Four stacked PCB modules for a working system build up to outer dimensions of about 40x40x40mm without lens and form an almost cubic IP camera.
The complete stack can be screwed on simple standoffs to a mechanically stable system and fastened in an outer housing.
Consuming power significantly less than 5 watts and featuring an operating temperature range of - 20 .. + 85 °C a fan cooling system is dispensable in most cases (simple heatsink instead), so that small box camera designs can be designed.
Incidentally, the S5L μSoM system also offers the option of designing "two-eyed" cameras - since the S5L supports 2 image sensors on its Vin-port, 2 Flex connectors are implemented on the SoM carrier board (not shown). Thus, cameras with 2 identical image sensors are possible, e.g. for 270°...360° monitoring or with 2 different sensor specifications, e.g. for optimized day/night use.
Image warping (or de-warping in its inversion) is an image transformation algorithm in which all positions of the pixels in the source image are translated to new positions in the target image. All other pixel parameters such as color or brightness values remain unchanged. For example, de-warping is used in document scanning to compensate for optical distortions, but it is also used to render extremely wide-angle optics (aka fisheye) because they naturally "bend" straight lines the farther they are from the centre, which is hard to watch from human eyes.
The correction of such extreme (180°) wide-angle shots has become a standard tool with increasing use in the video surveillance and automotive sector but requires an elegant implementation in order not to waste essential processor resources. In all Ambarella SoCs, de-warping is done in the image pipeline on embedded DSPs and therefore does not need resources from other functional areas. Streaming with 4kp30, H.264 encoding and fisheye de-warp will not generate more than 5% CPU load.
The example shown in figure 8 demonstrates a recording with 4k resolution at 30 fps, encoded with H.264 at 8 Mbps data rate using a Sony imx377 image sensor and a 180° wide-angle lens (top). Typical for footage taken with fisheye lens is the almost circular image. The de-warping profile "simple" was activated with maximum de-warping strength (middle). Image on bottom finally shows use of a digital zoom with factor 2 – as camera still has access to entire source image user can move the RoI in the zoomed image with electronic pan-tilt function and thus build a virtual PTZm camera.
Unlike Ambarella, the company Teknique is currently widely less unknown even to industry insiders. The young company is a specialized Ambarella® Solution Partner located in Auckland, New Zealand, with core expertise in image processing, optics, AI & Analytics. As such they have become adept in the design and engineering of custom camera solutions, modules and systems.
Teknique manufactures the Oclea™ System-on-Module (SoMs) family of products based on Ambarella System-on-Chips (SoCs) drawn from Teknique's 15 years of experience in vision products. Oclea™ SoMs enable faster time-to-market with reduced design cycle-time and risk, along with rich features such as extensive ISP functions and CVFlow™ CNN acceleration.
The currently available S5L-series μSoMs will soon be extended with the CV22 family, enabling CNN-based video analytics applications
|||3D motion-compensated temporal filtering noise reduction, see: https://getsafeandsound.com/2018/09/2d-3d-noise-reduction-security-cameras/|
|||Gstreamer Project landing´s page: https://gstreamer.freedesktop.org/|
|||See also: http://z25.org/static/_rd_videostreaming_intro_plab/index.html|
|||At the time of writing Teknique released r.0.9.4 as an intermediate version|
|Scripted Streaming - Application Report||631.54 KB|