The objective of this workshop is to explore the capabilities of a Raspberry Pi, utilize computer vision to detect objects by color, and integrate servo motor control with the color detection output.
A Raspberry Pi is a single-board, small computer that is used for electronic projects or as a component in larger systems (Figure 1). The default operating system is Raspberry Pi OS, a Debian-based operating system. A Raspberry Pi can also operate using Linux distros or Android. Internally, a Raspberry Pi has a central processing unit (CPU), graphics processing unit (GPU), and random-access memory (RAM) and can perform more advanced tasks, such as character recognition, object recognition, and machine learning, compared to Arduino.
The base hardware of a Raspberry Pi 4 features USB 2.0/3.0 ports for peripherals, a wireless and ethernet port, Bluetooth, USB-C power, general-purpose input/output (GPIO) pins (Figure 2), an audio jack, two micro HDMI ports, and a camera module port.
A Raspberry Pi has 28 GPIO pins, two 5V pins, two 3.3V pins, and eight ground pins. The 28 GPIO pins are analogous to Arduino digital/analog I/O pins and allow connections to sensors and additional components, such as an Arduino board.
The data storage for a Raspberry Pi uses an SD card. Many programming languages can be used on the Raspberry Pi, including Python, Scratch, Java, and C/C++. The Python programming interface is the default for a Raspberry Pi. Thonny IDE is the environment for Raspberry Pi when using Python.
To further the capabilities of a Raspberry Pi, additional modules, such as a camera module, microcontrollers, and programming libraries, such as the Open Source Computer Vision Library (OpenCV), can be implemented.
Python is a text-based programming language typically used for web development, software development, mathematics, and system scripting. Python is functional across several platforms and has syntax similar to the English language, which minimizes the amount of code needed to perform certain tasks compared to other languages.
Unlike other programming languages, Python does not require a closing mark on lines of code. Lines of code are closed off by continuing onto the next line.
In Figure 3, the program will print “Hello World”, make a variable, x, equal 3, make a variable, y, equal 2, add the two variables together, and store the result in a variable named result, and print the result.
For any loop made (if/elif/else statements, while loops, for loops), the lines following the initial declaration of the loop must be indented (Figure 4). The lack of indentation will cause a syntax error. In Figure 4, the program will determine if 5 is greater than 2, and if true, print “Five is greater than two!”.
A while loop executes the code within the loop so long as the condition is true. In Figure 5, the while loop will continuously print the result on a new line and add 1 to the result so long as the result is less than or equal to 10.
A break statement is used to break a loop regardless of the condition (Figure 6).
A for loop is used for iterating over a set sequence. The sequence can be a list, tuple, dictionary, set, or string. In Figure 7, the for loop will print each character in the string random_word with a space in between each character. Similar to the while loop, a for loop can be broken early with a break statement.
Comments within code are typically used to explain the code and make the code more readable and easier to troubleshoot when errors arise. Line comments begin with a # and can only span a single line. Block comments begin and end with a ’’’ and can span multiple lines (Figure 8).
Computer Vision (CV)
Computer vision is an area of artificial intelligence (AI) that allows computers and systems to measure data from visual inputs and produce outputs based on that data. The OpenCV library is an extensive open source library used for computer vision. OpenCV contains interfaces for multiple programming languages.
OpenCV allows image/video processing and display, object detection, geometry-based monocular or stereo computer vision, computational photography, machine learning and clustering, and CUDA acceleration. With this functionality, many applications are possible including face recognition, street view image stitching, and driverless car navigation.
In OpenCV, color images are represented as three-dimensional arrays. An image consists of rows of pixels, and each pixel is represented by an array of values representing its color in BGR (Blue, Green, Red) format. The array can be transcribed into the color’s specific hue saturation value (HSV). The HSV model describes colors similar to how the human eye tends to perceive color. This format is more suitable for color detection purposes.
cv2.cvtColor(frame, conversion code) is a method used to convert an image from one color space to another. In the OpenCV Library, there are over 150 different color-space conversion codes. The one used in this lab is cv2.COLOR_BGR2HSV, which converts an image from the default BGR format to the desired HSV format.
cv2.inRange(frame, lower bound, upper bound) is a method used to identify pixels in the frame that fall within the range of values defined by the lower and upper boundaries. This command outputs a binary mask where white pixels represent areas that fall within the range, and black pixels represent areas that do not fall within the range. The right panel in Figure 9 shows the output for the cv2.inRange() method using the command orange_mask = cv2.inRange(nemo,lower,upper). The bright orange pixels fall within the range specified and have a corresponding white value in the output.
cv2.bitwise_and(frame, frame, mask = mask) is a bitwise AND operation that delegates in making the frame only retain what is determined by the set mask. Only pixels in the frame with a corresponding white value in the mask would be preserved. The rightmost panel in Figure 10 shows the output of the command cv2.bitwise_and(nemo, nemo, mask=orange_mask).
cv2.bitwise_or(frame1, frame2) is a bitwise OR operation that delegates in combining multiple frames so that the output retains non-zero pixels from either frame.
cv2.imshow(window name, image) is used to display a given image in a window. The name of the window is given by the first parameter in the command. The window automatically displays in the same size as the image.
Numerical Python (NumPy) is a Python library used primarily for working with arrays. The library also contains functions for mathematical purposes, such as linear algebra, fourier transformation, and matrices. In this lab, the NumPy command used is np.array.
np.array() creates an array object of the class N-dimensional NumPy array, or numpy.ndarray. A list, tuple, or a number can be entered in the parentheses to turn the data type into an array.
Raspberry Pi Camera
The Raspberry Pi Camera Module is a camera that can be used to capture high resolution and high definition images and video. The module’s library features commands to use the camera within code and modify how the camera outputs the frame rate, resolution, and timing of the image or video it is capturing. The camera module’s coding library (picamera) is automatically installed onto a Raspberry Pi once the camera is enabled and the Raspberry Pi is rebooted.
PiCamera() is the function used to call the PiCamera within the code. Typically, PiCamera() is assigned to a variable to make calling the camera throughout the code easier.
PiRGBArray(camera, options) produces a three-dimensional array from a BGR capture. After declaring the camera, the Options parameter can be used to set the size of the image captured.
camera.resolution = (width, height) sets a specific resolution for the image captured by the module. The default image resolution is set to the resolution of the monitor being used. The maximum resolution for still photos is 2592×1944 pixels and 1920×1080 pixels for video recording. The minimum resolution is 64×64 pixels.
camera.framerate = frame rate sets a specific frame rate or the speed at which the image is shown. The default frame rate and maximum frame rate are dependent on the resolution at which the video or image is being captured.
camera.capture() enables the camera module to capture an image. The output file name, format type, video port usage, resizing, splitter port, bayer, and other parameters can be set.
camera.capture_continuous() captures images continuously until a defined end. For this lab, the format of the output is set to bgr, which means the image data is written to a file in BGR color space format. The output is a file-like object and each image is written to this object sequentially. The output must be cleared in preparation for the next image in the stream.
Raspberry Pi Setup
Before using a Raspberry Pi, the Raspberry Pi must be prepared with a formatted SD card with Raspberry Pi OS and the correct programming library packages installed. For this lab, these steps have been done. A more detailed explanation of the Raspberry Pi Setup can be found in the Appendix.
Servo Motor Setup
Continuous vs. Positional Servo Motor
A servo motor is a compact actuation device with high power output and energy efficiency. There are two kinds of servo motors used for common applications – continuous rotation and positional rotation servos. A continuous servo motor has a shaft that spins continuously allowing definite control over its speed and direction. A positional servo motor can only turn over a range of 180 degrees (90 degrees in each direction), allowing precise control over its position (Figure 11).
Programming Servo Motors (Raspberry Pi)
The following commands are used to program a continuous rotation servo motor once it’s wired to the GPIO pins on the Raspberry Pi:
GPIO.setmode(mode) is used to dictate how the GPIO pins on the Pi are referenced in the program. Following are the two options for GPIO mode:
- GPIO.BOARD – Refers to the pins by the physical position of the pin on the board
- GPIO.BCM – Refers to the pins by the Broadcom SOC channel number, which is an intrinsically defined system that differs for various versions of the Pi (refer to Raspberry Pi GPIO Pinout)
In this lab, we will be referencing the GPIO pins with the GPIO.BCM method.
GPIO.setup(servoPin, mode) sets the servoPin to either an input or output. The corresponding options for the mode are GPIO.IN and GPIO.OUT
PWM stands for Pulse Width Modulation. Servos use a pulse-width modulated input signal that defines the state of rotation or position of the servo shaft. GPIO.PWM(servoPin, frequency) initializes PWM to a servo pin and a certain constant frequency.
pwm.start(duty cycle) begins servo rotation with a specified duty cycle value, which is a measure of the speed and direction of rotation of the shaft. For the continuous rotation servos used in this lab, the shaft rotates:
- Clockwise for a duty cycle value of less than 14
- Counterclockwise for a duty cycle value of 14 or greater
pwm.stop() ceases servo motor rotation entirely.
GPIO.cleanup() resets the ports and pins to the default state.
Materials and Equipment
- Raspberry Pi with SD card
- Raspberry Pi Camera Module
- Monitor with micro HDMI to HDMI cable
- USB keyboard
- USB mouse
- Power supply
Ensure that the Raspberry Pi is connected to a power supply and a monitor. Verify that the camera module and the USB mouse and keyboard are plugged into their corresponding ports on the Raspberry Pi.
Part 1: Testing the camera module
In this part, capture a still image using the camera module to verify that the camera has been connected and enabled correctly.
- Open a terminal window by clicking the black monitor icon on the taskbar (Figure 12).
- To take a still picture, type in the following command raspistill -o testpic.jpg.
The camera should take a still picture and save it as testpic.jpg on the Raspberry Pi. A preview should be visible on the screen for a few seconds. If this command returns any errors, please notify a lab TA.
Part 2: Wiring the Servo Motor
- Wire the servo motor according to the following diagram. Make sure to wire the servo pin to GPIO 17 on the Raspberry Pi, as shown in Figure 13
Part 3: Color Detection using Pi Camera and Python-OpenCV
In this part, a Python script that isolates red and green colors from a live video captured by the camera module will be developed and this output will be used to create a prototype of a traffic light detection system.
- Download the Python script for color detection using OpenCV. Open the camera module with the Thonny Python Editor installed on the Raspberry Pi.
- There are missing components in the program. Complete the code following the instructions outlined in the next steps.
- The code is divided into four sections. Figure 14 shows the workflow for this lab.
- Section 1 includes commands for importing all necessary packages and setting up the GPIO pins on the Pi.
- Section 2 sets up the camera module so it captures a continuous video.
- Section 3 contains code for color detection and isolation for red and green colors.
- Section 4 integrates the servo motor with the color detection output such that the servo rotates clockwise when red is detected, counterclockwise when green is detected, and stops rotating when neither red nor green is detected.
Type the code in Figure 15 to import the required packages into the Python module.
- OpenCV represents images as NumPy arrays. Lines 3-5 import OpenCV, NumPy, and PiCamera libraries. The images obtained from the camera module must be in an array format so Line 6 is included.Line 7 imports the Raspberry Pi GPIO library.
- Assign a variable for the servo pin. Based on the wiring diagram, the servo should be wired to pin 17 on the Raspberry Pi.
- Set GPIO mode to GPIO.BCM
- Set the mode for the servo pin as output. This completes section 1 of the code.
- Section 3 will perform color detection for red and green colors. Since it is easier to identify these colors with their HSV, each frame is converted from the default BGR format to the HSV format. Line 17 in Figure 16 performs this task using the cv2.cvtColor method. The newly created HSV frame (stored as variable hsv) will now be used for color detection.
- The segment of code in Figure 17 will isolate red from the video. Lines 20-21 define the lower and upper boundaries of the range of HSV values within which the color would be classified as red. These values might have to be adjusted based on lighting and exposure conditions.
- Line 22, identifies all the pixels in the video that fall within the specified range by defining a variable red_mask, and using the cv2.InRange method using red_lower and red_upper as boundaries. This line outputs a binary mask where white pixels represent areas that fall within the range, and black pixels represent areas that do not.
- Line 22 applies the mask over the video to isolate red using the cv2.bitwise_and method taking frame as the input, and outputting frame using red_mask as the mask. This line outputs the frame so that only pixels that have a corresponding white value in the mask are shown.
- To display the final result (Line 34), type a line of code that creates a new window named “Red and Green Detection in Real-Time” and displays the final output. Use the cv2.imshow method.
- Note that Line 36 is required because it stops rawCapture to clear the stream in preparation for the next frame.
- This concludes Section 3 of the program. Get the code approved by a TA and move to Section 4.
- Lines 45-61 represent the final section of the program. The objective of this section is to integrate the color detection output with servo rotation. The servo would rotate clockwise when a red object is detected, counterclockwise when a green object is detected, and would stop rotation when neither red nor green is detected.
- Lines 46-47 dictate that every time the key ‘q’ is pressed, the camera closes and stops capturing new frames. The final frame is checked for the presence of red and green using if statements (Lines 48-58).
- The following conditions are implemented for detecting red color (line 48)- the frame should have more red pixels than green, and the number of red pixels should be at least 25% of the total number of pixels. The cv2.countNonZero method is used on red_mask to obtain the number of red pixels.
- On line 50, insert a command to start servo rotation with a 5% duty cycle. This corresponds to clockwise rotation.
- Insert the following conditions for detecting green color (line 52)- the frame should have more green pixels than red, and the number of green pixels should be at least 25% of the total number of pixels.
- On line 54, insert a command to start servo rotation with a 55% duty cycle. This corresponds to counterclockwise rotation.
- On line 58, insert a command to stop servo rotation.
Source: RAD Workshop