Analyze sounds using AI on the edge and fleet intelligence.
Can we use AI to help us identify and quantify noise pollution? In this project, we’re using a Coral Dev Board or a Raspberry Pi 4 with an attached Edge TPU to listen to ambient sounds 24/7 and categorize them for us.
We’ll also demonstrate how we could deploy a fleet of these devices (e.g. across an entire city) and feed new information back to all the devices to make them “smarter.”
Contents
Before you start
- Overview
- Hardware requirements
- Software requirements
Tutorial
- Setting up an edge node
- Add a device and download the balenaOS SDK image
- Flash the SD card and boot the device
- Using the noise analyzer
- Using the web interface
- Uploading files
- How it works
- What’s next?
Overview
To help reduce noise pollution, we first need to know which noises are most prevalent and how often they occur. Is it dogs barking at night or loud machinery during the day? How often does it actually occur? Is it worse in one part of town or another?
The use case is not limited to noise pollution. We can train our sound categorization model to recognize almost any audio we want, assuming we have enough samples of the target sound. Sounds we can identify and quantify could include:
- a roadside installation could detect failing truck brakes based on their audio signature
- an auto repair chain could detect various engine sounds indicating different mechanical problems
- property management companies can proactively try to detect faulty machinery and hardware (e.g. HVAC fans and systems) on rooftops
- and many more examples…
What is an Edge TPU?
TPU stands for “Tensor Processing Unit” and is hardware that accelerates machine learning tasks. The “Edge” TPU is a much smaller, more power-efficient version of a TPU that is designed to run on edge devices. The Coral Dev Board has a built-in Edge TPU but you can also get an external one such as the USB Accelerator that can be plugged into a Raspberry Pi. The Edge TPU speeds up inferencing (classifying data to infer a result) through its hardware which is optimized for TensorFlow Lite.
Our setup
Our edge device, or node, will be either a Coral Dev Board or a Raspberry Pi 4 with an attached Edge TPU. The nodes will use an on-board or external microphone to constantly listen for noises above an adjustable volume threshold. Any detected noise will be recorded as a WAV file, processed, and fed to our TensorFlow Lite model running on the local device’s Edge TPU.
Learn more about how balena has partnered with Coral to provide local AI projects, like this one, to give edge developers a strong hardware and platform start to their projects.
Our model will make a prediction as to the type of the sound, which we can view on a web page hosted on the device. There we can decide if the prediction was accurate, and if not, describe the sound and upload it back to a “master node” so it can be used to eventually re-train the model. The re-trained model can then be distributed back to the nodes, helping the entire fleet to make more accurate predictions.
The optional master node will be an amd64 device (such as a NUC) which will process our sound files, train them on a Tensorflow model, and then convert the model into one that can run on the Edge TPU. We’ll then transfer that updated model to our fleet of “nodes.”
We’ll use balenaCloud to manage the master node and the edge nodes, as well as all communication among the fleet.
Hardware required
All you need to get started is one compatible edge device (see below) on which you can install the application to start identifying sounds. (We’ve included a pre-trained model that identifies ten common city noises.) Part 1 of the post concentrates on this type of setup.
- Coral Dev board and/or a Raspberry Pi 4 with a Coral Edge TPU and microphone (Raspberry Pis don't have a built in mic)
- SD card for Coral Dev board or Raspberry Pi.
- USB-C power adapter for the Pi 4 or Coral Dev
Software required
We’ve written all of the software already, so you’ll just need to use the “Deploy with Balena” button below to create an image you can burn to an SD card. We recommend balenaEtcher to burn your images.
You can check out the balena GitHub repo for the audio analyzer project if you're an experienced user and want to deploy this project manually with balenaCLI. If you plan to follow part two by setting up a master training node, you may find it more convenient to clone this project's repository locally and use the CLI.
Setting up an edge node
We’ll start by setting up one edge node that will use a pre-trained Edge-TPU compatible model to identify 10 different noises based on the Urban Sound Dataset. These noises include: air conditioner, car horn, children playing, dog bark, drilling, engine idling, gun shot, jackhammer, siren, and street music.
You’ll need a free balenaCloud account (your first ten devices are free and fully-featured) to set up and manage your edge device. Once you’ve created your balenaCloud account, and are logged in, you can use this link to deploy to balenaCloud.
Select your device type (either Coral Dev or Raspberry Pi 4) and then click “Create and deploy.” Note: you can name the application whatever you choose to. We provide the repo name as a default.
At this point, balenaCloud is automatically copying all of the software for the project from the GitHub repo to your application and building it in the background.
Add a device and download the balenaOS disk image
Next, add your device to the application by flashing your SD card. Within the application, select on the upper left corner ‘Add device.’ If your device has WiFi capability that you wish to utilize, be sure to select ‘Wifi + Ethernet’ to set up the wireless network. Enter your WiFi SSID and password – this injects the details into the downloaded OS so that the device will automatically link up to the WiFi when it starts up.
Flash the SD card and boot the device
Once the disk image is downloaded, you need to flash it into an SD card. We recommend using balenaEtcher to do so. Etcher will prepare a bootable SD card and validate that it was flashed correctly. Note that you can burn several SD cards with the same image file, and all the devices will boot and provision into your application's fleet.
If you are using a Pi 4, make sure the Edge TPU is plugged into one of the USB 3 (blue) ports, and a suitable audio input is available. (more on that below.) Then power up the device. If you are using the Coral Dev board, you’ll need to follow a few additional steps:
Unplug the Coral Dev Board and change the boot mode switches to 1:on 2:off 3:on 4:on to boot from the SD card. Insert the SD card into your Coral Dev Board and then power on the board using a 2-3A power cable connected to the USB-C port labeled “PWR”.
Warning: This will completely erase the internal storage media, so make a backup first if you've been using your device and have stuff you don't want to lose.
When flashing is complete, your board will shut down. Unplug the power, remove the SD card, and reset the boot switches to eMMC mode setting them to 1:on 2:off 3:off 4:off. Then power on the Coral Dev Board again to boot the device from eMMC. It will take a minute or two for the Coral Dev Board to appear on your balena dashboard.
After a few minutes, (or longer depending on your internet connection speed) you should see the application’s three three containers in a running state on the dashboard:
Now you are ready to classify some noises!
Start analyzing noise
Using the noise analyzer
The first step is to get some audio into the device using a microphone. The Coral Dev board has an on-board microphone which is set as the default, so if you want to use that, you don’t need to change anything. On the Raspberry Pi 4, (or the Coral Dev if you don’t want to use the onboard mic) you’ll need to use additional hardware.
A simple, inexpensive USB microphone such as this one is a quick way to get audio into the device. Beware that some inexpensive microphones have very poor sensitivity. It may be worth it to spend a little extra for higher quality. Another option is use a sound card, Pi HAT or outboard analog to digital converter such as a FocusRite iTrack Solo.
When the “recorder” container starts, it generates a list of detected audio input devices as well as the board’s default audio input. To view this information, click the “device detail” link at the bottom of the device’s web interface, which is discussed below.
If you want to change the input device from the listed default, you can add a service variable named INPUT_INDEX to the “recorder” container to choose a different input. To add a service variable, click the “Device service Variables” button on the left side of the dashboard, then “+ Add variable” at the top.
Source: AI Noise Analyzer
