Automatic Number Plate Recognition System

Introduction

Our project aims to design a parking barrier modeling system with automatic number plate recognition.In our project, we used two Raspberry Pi. One for controlling the car stuck with printed number plate to move linearly toward the barrier and stop if it is not allowed to enter. The other is for controlling two Infrared IR sensors to trigger camera and servo respectively, and for image processing and recognition. The whole parking system runs as shown in the figure below: when the car moves close to the barrier within a certain distance, a sensor will react and transfer a signal to the RPi. Then, RPi will control the camera to take a photo, and processing the image get from Rpi camera using openCV and transfer individual number or letter images to a sequence of characters using OCR so that the input number plate can be checked whether matched the stored number plates string in “database”. After processing with OCR, the number plate will be recognized by RPi, then displayed on the PiTFT. If it matches with the number plates in the if-statement we set before, the barrier will lift up to let the car in, or the car will be barriered out. After the car passes through the barrier and reaches a certain distance, another sensor behind the barrier will react and transfer a signal to RPi. And RPi will control the barrier to lift down.

Objective

• Raspberry Pi camera configure and control
• Image processing including number plate localization, background remove, character enhancement, noise remove and skew correction
• OCR recognition
• IR sensors configure and control
• Standard servo configure and control with barrier
• Pygame display
• Car control

The general idea is that several car number plates are prestored, meaning car with these number plates are allowed to enter the park (barrier will open) while other number plates not matched the system “database” will not be allowed entering (barrier keeps closed). Additionally, the car moves automatically with changeable number plates to demonstrate the whole function of the system.

Design & Testing

• Big Picture

• Hardware design

The hardware design of our parking system consists of two Raspberry Pi, a station, a barrier, two IR sensors , a modeling car and a camera. We first started with building the fundamental subsystem composed of a RPi, a camera, two IR sensors and a servo which is used to control the barrier.

For the camera, we connected it with RPi using a cable with proper length.

For IR sensors, We used GP2Y0D810Z0F, which is a distance measuring sensor to detects objects between 2 cm and 10 cm (0.8″ and 4″) away, composed of an integrated combination of PD (photo diode), IRED (infrared emitting diode) and signal processing circuit.For proper installation and use, we first soldered the pin portion and main sensor portion together, then, connected them with RPi respectively. Among three pins of each sensor, the left one needs to be grounded. The middle pin should be connected to the power supply (3V3 output on RPi), while the right one should be connected to a 1k resistance, then to GPIO to provide control signal. We chose GPIO19 and GPIO26 respectively as input pin since they are free available. When the sensor not detecting any object it input a high signal (“1”) into Pi, while once it detect an object a low signal (“0”) will be imported.

As For the servo, we used Parallax Standard Servo (#900-00005) because it is controlled by position rather than impulse time while encoding, and it can hold any position between 0 and 180 degrees.In our project, we needed the barrier lifted 90-degree up and lifted down to the initial position. Thus, the standard servo satisfied our expectation, and the quick-start circuit shows as below (using GPIO13 as output to control the servo) :

Besides the above subsystem, we also made a barrier using Cast Acrylic material and laser cut technology. Cast Acrylic is a thermosetting plastic with a superior surface finish, flatness and optical properties. It is also a hard and brittle material which is sensitive to stress concentrations. This is most evident if parts are small or thin, thus is proper to be made into a barrier. And we chose laser cutting mainly because of its easier workholding and reduced contamination of workpiece (since there is no cutting edge which can become contaminated by the material or contaminate the material). Precision may be better, since the laser beam does not wear during the process. There is also a reduced chance of warping the material that is being cut, as laser systems have a small heat-affected zone.Based on these, we designed a barrier model in CAD, which is 25cm long, drilling holes in the middle to fix camera, and also drilling holes on the one side to be connected with servo, as shown in the following figure:

Then, we designed a station to integrate the above parts into a whole. We used a paper box to accommodate battery with carrier, power bank and breadboard, and stuck the PiTFT on the top of the box to make the display clear to be seen. In addition, we used a plank to fix the position of two IR sensors and the paper box. Since the speed of the car is almost fixed, and the time cost of image processing and recognition will not vary significantly, we can determine the distance from the first sensor to the paper box to ensure enough time for RPi to analyze and control servo. In the same way, we can also determine the distance from the paper box to the second sensor to guarantee the whole car to pass through before the barrier lifts down.

As for the car, we used the one we assembled in previous lab. And we printed four different number plates with bold font and bold border for easier localization and recognition as follows:

To make the car move linearly, we attached a second Raspberry Pi on it and connected them with two Parallax Continuous Rotation Servos (#900-00008). The quick-start circuit is same as standard servo (we chose GPIO12 and GPIO13 as output of control signal), while the difference is that continuous rotation servo is controlled by impulse time.We used hardware PWM method to control two servos since they are not exactly same, it is inconvenient to set frequency or duty cycle to make them rotate with same speed. Therefore, we just gave same calibrate signals to these two servos, and used a screwdriver to adjust the rotation speeds as consistent as possible.

• Software design

Image capture use Raspberry Pi Camera

• After Raspberry Pi Camera was assembled and configured on the Pi, python code can be written to control the camera. First of all, PiCamera library need to be imported and initialized:
• from picamera import PiCamera camera = PiCamera()
• In addition, camera settings such as resolution, framerate, brightness, and rotation can be adjusted according to condition.
• Then, preview on monitor and image capture can be done by using
• camera.start_preview() sleep(2) camera.capture(‘image.jpg') camera.stop_preview()
• It should be noticed that, before image capture at least 2 seconds sleep time has to be introduced to give the sensor time to set its light levels. Also, camera preview only works when a monitor is connected to the Pi, so remote access (such as SSH and VNC) will not allow you to see the camera preview.

Image processing

• After image captured by pi camera, we need to process the image including number plate localization, background remove, noise remove, character enhancement and skew correction. In process result is shown as four figures below.
• To do these image processing in python code, opencv module is required. Therefore, we first install opencv on raspberry pi by running command:
• sudo apt-get install libopencv-dev python-opencv
• Then use import cv2 to import opencv module into python code.
• Detailed steps to process the image can be explained using the pseudo code below:
• number plate localization, background removeRead in captured image; Change RGB to grayscale; Noise removal with iterative bilateral filter(removes noise while preserving edges); Histogram equalisation for better results; Morphological opening with a rectangular structure element; Image subtraction(Subtracting the Morphed image from the histogram equalised Image); Thresholding the image; Applying Canny Edge detection; Dilation to strengthen the edges; Finding Contours in the image based on edges; Creating the kernel for dilation; Finding Contours in the image based on edges; Sort the contours based on area; Loop over our contours: Approximate the contour with 6% error; Select the contour with 4 corners as the number plate’s counter; Obtain the 4 corner position, achieve number plate localization; Masking the background as black background Segment the number plate region from the whole image according to 4 corner coordinates;

• Character enhancement and noise revomentThresholding the segmented grayscale image again to black and white; Created a structuring elements as kernel; Morphological opening(erosion followed by dilation) with a rectangular structure element to remove noise;

• Skew correctionFlip the foreground and background to ensure foreground is now “white” and the background is “black”; Grab the (x, y) coordinates of all pixel values that are greater than zero; Compute a rotated bounding box that contains all coordinates; Obtained the rectangle rotating angle (in range [-90,0)) using cv2.minAreaRect(); If rectangle rotates clockwise add 90 degrees to the angle; else just take the inverse of the angle to make it positive; Rotate the image matrix according to the angle obtained above

After above image processing steps, the result image is like following figure shown and is clear enough to be passed for recognition. Therefore, the aim of image processing achieved.

Number plate recognition

In this project, Optical Character Recognition(OCR) engine was used for recognition, that is transfer the characters in image to text. In this way we can check whether the character on number plate is same as the one stored in database to decide whether it is an legal number plate allowing entrance.

In order to use the OCR engine, we first installed the open source tesseract-ocr package which contains OCR engine -libtesseract and command line program -tesseract by running:

sudo apt-get install tesseract-ocr

After that, we can use command line in format below to run tesseract for image recognition:

tesseract imagename outputbase [-l lang] [--oem ocrenginemode] [--psm pagesegmode] [configfiles...]

In this project we used command line below to recognize image processed result “output2.jpg” and store the recognition result into “file.txt” with specifying the language as english and page segmentation modes as treat the image as a single text line:

tesseract output2.jpg file -l eng -psm 7

Then, result in txt file can be read out for following logical control.

Logical control and display using pygame

The whole system logic is that when the front end IR sensor detect the car it will trigger the pi camera to take a photo and then Raspberry pi process the image and user OCR engine to recognize the number plate and then check the recognition result with those stored in database, if the number plate matched with one in database, this is a legal number plate, servo will be controlled to lift up parking barrier to allow car entrance. At the same time, the recognition result will be displayed using pygame on PiTFT as well as the image that captured by pi camera and a message said “Allow pass”. And then, when car leaving and detected by the second IR sensor, servo will be triggered to lay down the barrier and now the system welcomes the next car coming for check with PiTFT displaying a welcome screen. However, if the recognized result was found that this is an illegal number plate, then the servo will be controlled to not lift up the barrier, and on PiTFT, instead an allowance message but a message said “Not allow” will be displayed so that “driver” need to stop the car and leave. After wait for a few seconds, the whole system welcomes the next car coming for check.

Python to achieve above whole system logic can be demonstrated by the two logic flow chart below:

The first chart is the logic flow chart for whole system outer while loop

The second one is logic flow chart for servo control function inner while loop

Hardware PWM for servo

• In order to get more stable signal, we decide to use hardware PWM method to both control the standard servo and continuous rotation servos.
• After import pigpio library, we setup piTFT buttons for broadcom numbering first. Then, connect to pi gpio daemon:
• pi_hw=pigpio.pi()
• For the standard servo, it is controlled by position. According to the datasheet, we set the initial place at 0.00225 by code below:
• pi_hw.hardware_PWM(13,f,dc)
• 13 stands for GPIO13, used as PWM output. f is frequency, equaling 1/(0.00225+0.02) since the servo needs to receive a pulse every 20 ms in order to hold its position. And dc is duty cycle, equaling (0.00225/(0.00225+0.02))*1000000.
• When the car with legal number plate comes nearly, the servo should drive the barrier to lift up to 90 degree. It can be realized by setting the position at 0.00125, thus the frequency and duty cycle also change, controlling the servo to hold almost 90-degree position. In the similar way, just set the position at 0.00225 to control the servo to lift down after car passing. It should be noticed that after each time setting a new position, frequency and duty cycle should be both set to zero to initialize the servo, making it ready for receiving next position signal.
• For continuous rotation servos, we choose GPIO12 and GPIO13 as PWM outputs respectively. The controlling code is same as above.
• Noticed that we should run command sudo pigpiod before running python codes in order to launch the pigpio library as a daemon and use it correctly.

---

Issues & Solutions

• Reduce image processing time

In this project, in order to have more accurate recognition result, we set the resolution of pi camera to a relatively large value 2592*1944, but this will cause the image processing time become long (13s). Our solution to this problem is to segment the image into smaller one containing the number plate after image capture but before image processing. This method is feasible because position of numblate on car is fixed, position of camera on barrier is also fixed, hence the position of number plate on the captured image is also relatively settled. And this method can reduce the image processing time to around 5s.

• Run OCR engine for recognition

• At the testing stage of this project we imported ocr pytesseract module in order to run ocr engine in python code.
• We install it by running:
• sudo pip install pytesseract
• And applying it in python file by using:
• from PIL import Image import pytesseract im=image.open(“image.jpg”) text=pytesseract.image_to_string(im, lang='eng')
• This module works well when we not use “sudo” to run this file. However, after we added pygame part and want to use “sudo” to execute the python code to display on PiTFT, an error message shows no pytesseract library. We assume this is due to under the admin access it cannot find the path to this library. Therefore, we came back to the original method to run ocr engine using command line, store the result to file and then read it from the file for following process. To run this command line from python code we import subprocess:
• import subprocess cmd=”tesseract output2.jpg file -l eng -psm 7″ print subprocess.check_output(cmd, shell=True)
• In this way, this python code can be executed by “sudo”. And this solution is inspired by the idea in 2018 spring project “The Eye”.
• Make car running linearly

In order to run the car linear, we at first apply symmetric duty cycle to GPIO12 and GPIO13 to get PWM. However, even though the hardware PWM signal is stable, two servo rotating with different speed due to these two servo is not identical. And because we want to run the car at low speed, the servo speed is highly affected by hardware condition of servo, hence, if we use software to adjust the duty cycle to achieve approximate same speed of two servo, this point can easily changed when restart the servos. Therefore, we finally decided to use hardware PWM to provide stable calibration signal (high pause 1.5ms), and use screwdriver to calibrate the two servo to approximately same speed. This method is more efficient than calibrating using software.

Source: Automatic Number Plate Recognition System