DeepPicarMicro Crams NVIDIA’s PilotNet Autonomous Vehicle Neural Network Into a Raspberry Pi Pico

Clever optimization approaches take a CNN model designed for high-end GPUs and run it on the low-cost RP2040.

A trio of scientists from the University of Kansas have published a paper on DeepPicarMicro, an autonomous vehicle testbed, which crams a fully-functional convolutional neural network (CNN) onto a Raspberry Pi Pico microcontroller board.

“Running deep neural networks (DNNs) on tiny Microcontroller Units (MCUs) is challenging due to their limitations in computing, memory, and storage capacity,” the team admits. “Fortunately, recent advances in both MCU hardware and machine learning software frameworks make it possible to run fairly complex neural networks on modern MCUs, resulting in a new field of study widely known as tinyML. However, there have been few studies to show the potential for tinyML applications in cyber physical systems (CPS).”

That’s where DeepPicarMicro comes in: a cyber physical system testbed for an autonomous radio-controlled model car based around a Raspberry Pi Pico and its RP2040 microcontroller. While far from the lightest-weight microcontroller around, that poses a challenge: the RP2040 has just two process Arm Cortex-M0+ processor cores running at a default speed of 133MHz and 264kB of static RAM (SRAM), far below the specifications of a typical autonomous vehicle edge AI system — and particularly the PilotNet architecture, developed for use with NVIDIA’s high-end resource-rich autonomous vehicle platforms, which the team sought to port.

To get PilotNet running on the Raspberry Pi Pico, it had to be optimized for the RP2040. For this, the team used a series of techniques beginning with eight-bit quantization, which reduced the layer memory requirements to 100kB or less for a drop in accuracy from 87.6 per cent to 86.9 per cent — but saw per-frame processing times jump to three seconds, far to slow for an autonomous vehicle system.

Scroll to Top