Advanced Driver Assistance Systems (ADAS) can deliver solutions that can meet the demands for greater road safety and better travelling experiences for drivers and other vehicle occupants. Systems like lane departure warning, autonomous braking and park assist are widely available on current models, and even more powerful features such as lane keeping, traffic jam assist, and adaptive cruise control will combine to enable fully self-driving vehicles in the future.
Many of the systems in today’s vehicles utilize machine vision based on conventional signal-processing techniques to detect and identify objects. Deep Learning Neural Networks have opened up exciting avenues of research for the automotive industry, which is keen to further improve and extend ADAS capabilities. This technology points the way forward for an industry looking for faster response and greater recognition accuracy in order to realize autonomous driving ranging from short-duration assistance modes like highway autopilot all the way to full-time driverless travel.
The automotive industry, led by major brands, is investing in the technology and aligning with leading computing businesses and centers of competence including Silicon Valley and academia. Its involvement has been a catalyst for the development of neural networks suitable for deployment in an embedded system, which is needed to meet the constraints on system size, cost and power consumption needed for the automotive environment.
Lightweight Embedded Neural Networks
Implementation of convolutional neural networks (CNNs) can be broken down into three main phases: training, translation, and the execution of CNNs into a production-ready solution. The utilization of systems that are most advantageous for each phase is necessary to achieve a cost effective, efficient result targeted for high-volume vehicle applications.
Training is often done offline with the use of CPU based systems, graphics processors (GPU) or Field Programmable Gate Arrays (FPGAs). These systems are optimal for training because of their high computational attributes and familiarity among design professionals.
In the training phase, developers utilize frameworks such as Caffe to model and optimize CNNs. A reference image database is used to determine the optimum weights for neurons in the network. Upon completion of the training, the conventional approach is then to generate the network and prototype on CPUs, GPUs or an FPGA, typically executing floating-point arithmetic to ensure optimum precision.
This approach has some critical drawbacks as a solution for use on-board vehicles. Limitations on computational efficiency and high cost prohibit them from being used in high-volume production systems.
CEVA has demonstrated an alternative approach that alleviates the loading imposed by floating-point arithmetic, and enables real-time performance at a power level that is acceptable for automotive use. With the further explosion of computation needed for fully autonomous driving, having a strategy to accelerate key functionality is imperative for the wide scale realization of these systems.
The revised strategy for network generation, using a framework called CDNN, takes a trained network structure and weights, developed on a high-power floating-point computing platform using a conventional network generator such as Caffe, and converts it to a slim, customized model based on fixed-point arithmetic. This model is then targeted to run on a low-power embedded platform based on an optimized imaging and vision DSP chip. Figure 1 illustrates the workflow for generating the lightweight embedded neural network. This approach brings high-performance neural processing within the modest power budget of today’s production vehicles, while degrading image-recognition accuracy by less than 1% compared to the original network.
Figure 1. CDNN translates conventionally generated network weights to a fixed-point network.
A 24-layer convolutional neural network with 224×224 input size and 11×11, 5×5 and 3×3 convolution filters, hosted on a low-power embedded platform, has demonstrated almost three times the performance of a comparable CNN implemented in a typical hybrid GPU/CPU processing engine, while requiring only about one-fifth of the memory bandwidth and a significant reduction in power consumption.
The Next Generation
Gained experience by the automotive industry into neural networks continues to move the technology forward, developing more advanced network architectures and increasingly complex topologies such as multi-layer per level topologies, multiple-input/multiple-output and fully convolutional networks. New and important network types are emerging to not only identify objects, but to recognize scenes and provide the image segmentation needed to address the applications in the automotive sector such autonomous driving functionality.
The automotive industry is simultaneously learning from academic discoveries, as well as driving development of automotive use cases and working to accelerate deployment of machine-learning neural networks for embedded systems.
Enhancements to the CNN network generator have added support for the new network architectures and topologies, such as SegNet and others like GoogLeNet and ResNet, as well as advanced network layers (figure 2). In addition, converting the pre-trained networks to an optimized real-time implementation is faster and easier with pushbutton activation. To ensure support for popular neural network generators, the CDNN framework works with both Caffe and TensorFlow, Google’s software library for machine learning.
Figure 2. Network-generator evolution introduces support for new layers and deeper architectures.
Embedded deployments can also take advantage of tremendous recent gains in scalability and flexibility that are reflected in recent updates to optimized embedded processing platforms. As more evolution continues to occur with the field of deep learning, it is important to have an architecture that can handle the demands of today’s processing needs, but also have the flexibility to adapt to the innovation that is certain to come in the future.
Getting Road Ready
This first wave of neural network applications will be focused on visual processing to support features such as automatic pedestrian, traffic-sign or road-feature recognition. As the performance of these systems continue to advance, for example to handle increasingly large data sets received from high-resolution cameras, neural networks are also expected to take on further roles in future generations of automobiles. These will include other complex signal-processing tasks in systems such as radar modules and speech-recognition systems.
As neural networks find themselves in the first production on-board autonomous driving systems, reported to be introduced on new vehicles launching in 2019-2020 model years, there becomes more demand for systems that are both safe and reliable. To finalize such systems ready for consumer use, automakers must also ensure compliance with applicable safety standards such as ISO 26262 functional safety, which encompasses hardware, software and system-level development.
These systems are becoming ever more complex and the need to ensure reliability and safety along with the processing demands becomes an increasing challenge for automakers.
Conclusion
Machine-learning neural networks continue to trace an aggressive development path that challenges efficient processing capability. State-of-the-art neural network architectures are already demonstrating recognition accuracy exceeding human performance. The latest frameworks for network generation, such as CDNN2, are driving the development of lightweight, low-power embedded neural networks that will deliver high accuracy and real-time capability in current advanced driver-assistance systems.
The first deployments for production vehicles will target basic visual-recognition systems, and ultimately will extend to support numerous complex signal-processing challenges in increasingly autonomous vehicles of the future.