Smart Door System based on raspberry pi

Introduction

The following subsections have an overview, motivation, importance, objective, problem analysis, list of requirement and expected results.

1.1     Overview of the project

At any moment, you may have a visitor, that visitor may be a friend who has something to say or an intruder who might make some problems for you, so what if that visitor comes when you are away from your home.

Our project aims to help you to know who visit you, which day and at what time they came, and you will be notified at the time when the visitors arrived, so that you can decide what to do then.

We solve that problem mainly by using Raspberry Pi. Our project also uses a sensor to detect human movement from a relatively long distance. Moreover, it contains a camera to capture images for the visitors.

If there is someone in front of the door, the sensor will send a signal to the Raspberry Pi that sends a signal to the camera to take an image. Then it compares that image with family members faces stored in the system by face recognition algorithm if the visitor is identified as family member the system, then requires Radio Frequency Identification (RFID) tag as an additional identification method through provide unique identifier for the family members. Otherwise, it saves the image of that visitor along with the time, then it sends a notification to owner’s smartphone who can send feedback to the visitor, if the owners discover that the visitor is an intruder, he can notify the police by calling them to come to the referred home.

1.2    Motivations

The desire to reflect what we have learnt from the courses that we took in our field (Computer Systems Engineering). Such as hardware design related courses (Microprocessor Interfacing and Embedded Systems). As well as software, related courses (Object oriented, Data structure and Networks Programming).

Moreover, the desire to know who is visiting the house when you're outside, is an important thing for several reasons, such as security-related matters.

Furthermore, the final outcome of this project will be a profitable product. Since it’s intended to be used by a large customer segment.

1.3     Importance

Making users aware of any move around their homes when they are away from their homes. And provide quick and easy tool for dealing with any person arrived when the user is far away from his home. And made homes more secure through face recognition algorithm and RFID technology.

1.4     Objectives

The main objectives that we are looking to achieve in our project are the following:

• To build system that’s detect the visitor’s motion in front of the door and send a signal to the camera to capture an image for that visitor.
  • Enabling the home owner to browse events and open the captured images.
  • Enabling the home owner to record a voice message to communicate with the visitor.
  • Design the android app that will connect the user with the door system.
  • Protecting the home via using face recognition algorithm and RFID technology.

1.5     Problem Analysis

Many people always travel from place to place due to business demands. Some people spend a couple of days away from their home leaving all their household without any kind of monitoring and control.

Everyone wants to know who is at his front door, which day and at what time they came, to be notified when they arrived. Wants to communicate with him in simple and fast and easy way. The main problem in our project is how to apply face recognition algorithm to detect and recognize face to determine this face is family members or not.

1.6    List of Requirements

System requirements can be summarized as:

• We need to use Raspberry Pi as a single board computer, to controls all system functionalities.
  • We need to use RFID technology as an identification through smart cards to provide unique identifier.
  • We need to use speaker to play the voice messages for the visitor.
  • List of hardware equipment like Passive Infrared (PIR) sensor, and internet connection used to connect user’s smartphone with the system and pi-camera to capture image.
  • We need to use smart phone to notify the user, view images, and send feedback messages.
  • Raspbian operating system and python programming language: We will use Raspbian operating system for Raspberry Pi and python programming language.
  • Face recognition process should provide real-time and accurate results and check whether the visitor is family member or not.
  • Home owner application must be user-friendly and make user able to view notifications and send feedback messages.

1.7     Expected Results

The results that we expect to attain are:

• Home owner application notifies the user and provides him the ability to browse and view events.
  • Smart door system hardware saves the events and send notifications.
  • The system is able to recognize the face of the visitor and distinguish it.
  • The system is able to use Radio Frequency Identification (RFID) as an additional identification method.
  • The system is able to send feedback to the visitor as a voice message.

Background

Overview

This chapter introduces the literature review of the project, description of technologies to be used in the project and some description of hardware and software component used in the system, finally some description of design specification and constrains.

2.1     Literature Review

Ring video doorbell is similar product to our project, it’s connects to your home Wi- Fi network and sends real-time notifications to your smart phone or tablet when someone is at front of your door, using Ring application which is available for Apple, Android, and Windows 10 devices, you can see an HD video stream of the person at front of your door and speak to them using two-way audio communication. Ring doorbell can alert you when someone presses the button on your doorbell or when motion is detected. Internet Service Provider (ISPs) in United Arab Emirates, Qatar, Jordan, and Saudi Arabia block ring’s connection protocols. That means that you won't be able to use Ring Doorbell in in these countries [10].

Moreover, there is a similar project to our project has done by a group of our university students, the title of that project is “home automation system based in raspberry pi single board computer” [1].

The project main idea is to design and implement home automation application using Raspberry Pi and based on python environment. A special sensor is used to sense body motion and send a signal to Raspberry Pi if there is someone in front of the house. Consequently, the Camera captures an image for the visitor then starts recording a video. The images and videos are saved on the server. The image is sent to the home owner by Wi-Fi. Homeowner can send feedback to the visitor as a voice message. Homeowner and police can connect by the Q-python application on homeowner smart phone. If the visitor gets out of the range of sensor, the camera will stop recording video. The components of the project have been built and worked well after programming the code on Raspberry Pi and homeowner smart phone.

The most similar project to our project between the two projects is “home automation system based in raspberry pi single board computer”. However, our project different in many aspects like:

• User can record and send voice messages to the visitor.
  • Our system uses the face recognition algorithm to recognize the face of the visitor and distinguish him.

Our system uses RFID tags as an authentication method.

2.1     Technologies to be used in the project:

This subsection illustrates the main technologies to be used in this project and what task they do.

1.   Raspbian operating system:

“Raspbian is a free operating system based on Debian optimized for the Raspberry Pi hardware. An operating system is the set of basic programs and utilities that make your Raspberry Pi run. However, Raspbian provides more than a pure OS.” [2]. Raspbian is comes with over 35,000 packages, pre-compiled software bundled in a nice format for easy installation on your Raspberry Pi [2].

2.   Python Programming Language:

Python is an open source computer programming language that lets you work more quickly than other programming languages. Experienced programmers in any other language can pick up Python very quickly, and beginners find the clean syntax and indentation structure easy to learn [3], furthermore it was supported by OpenCV library that provide face recognition features in our project.

3.   OpenCV Library:

OpenCV is the most popular library for computer vision. Originally written in C/C++, it now provides bindings for Python.

OpenCV uses machine learning algorithms to search for faces within a picture. For something as complicated as a face, there isn’t one simple test that will tell you if it found a face or not. Instead, there are thousands of small patterns/features that must be matched. The algorithms break the task of identifying the face into thousands of smaller, bite-sized tasks, each of which is easy to solve. These tasks are also called classifiers.

The algorithm may have 30-50 of these stages or cascades, and it will only detect a face if all stages pass. The advantage is that most pictures will return negative during the first few stages, which means the algorithm won’t waste time testing all 6,000 features on it. Instead of taking hours, face detection can now be done in real time [3].

4.   Android Studio:

Android Studio is Android's official IDE. It is purpose built for Android to accelerate your development and help you build the highest-quality apps for every Android device. It offers tools custom-tailored for Android developers, including rich code editing, debugging, testing, and profiling tools [4].

2.1     Hardware Components of the System:

This part describes the components and devices that used in the system and how does integrated with each other by raspberry pi.

1. Raspberry pi 3:

“The Raspberry Pi is a low cost, credit-card sized computer that plugs into a computer monitor or TV, and uses a standard keyboard and mouse” [5].

We chose raspberry pi 3 for our project, because it has an integrated wireless LAN so we don’t need to use additional devices to connect our system by the internet, moreover we chose it because it provides high computing power with its 1200 MHz Quad-core processor.

More specifications are listed here:

1. 1GB RAM.

• 4 USB ports.

• 40 GPIO pins.

• Full HDMI port.

• Combined 3.5mm audio jack and composite video.

• Camera interface (CSI).

• Display interface (DSI).

• Micro SD card slot.

• Video Core IV 3D graphics core.

1. Ethernet port.

With these features for the raspberry pi, we can build all system features on it.

2.   Pi Camera.

The main function in our project is to identify who visit the home and that thing will never be achieved without taking an image for that visitor.

The  Raspberry  Pi  camera  module  can   be   used   to   take   high-definition video,  as  well  as  stills  photographs.  It’s  easy  to  use  for  beginners,  but  has  plenty  to  offer  advanced  users  if  we’re  looking  to  expand  our  knowledge.  There  are  lots  of examples online of people using it for time-lapse, slow-motion, and other video cleverness. We can  also  use  the  libraries  we  bundle  with  the  camera  to  create  effects. It attaches via a 15cm  ribbon  cable  to  the  CSI  port  on  the  Raspberry  Pi.  There  are  numerous  third-party  libraries  built  for  it,  including  the  Pi  Camera   Python library. The  camera  module  is  very  popular  in  home  security  applications,  and in wildlife camera traps. We can also use it to take snapshots [6].

3.   Audio playback.

We also need to communicate with visitors so that we need sound playback device to play voice messages.

The Raspberry Pi has an on-board audio jack, which is super handy for all kinds of sound effects and speech We use Raspberry Pi jack to connect with speaker to output the voice message from home owner to the visitor. The voice message recorded by home owner smart phone using Android Studio application.

4.   PIR Sensor.

Referred to “Passive Infrared sensor” is an electronic sensor that allow you to sense motion, almost always used to detect whether a human has moved in or out of the sensors range. They are small, inexpensive, low-power, easy to use and don't wear out. For that reason, they are commonly used in home security. Sensing range is about 7 meters (120- degree cone), Runs on 5V-16V. Digital signal output is 3.3V [7].

That device will trigger the system with who in front of the house.

5.   RFID

Radio-Frequency Identification is an additional identification method at the same purpose as a bar code or ATM card, it provides a unique identifier for that object, and the RFID device must be scanned to retrieve the identifying information through smart cards to allow authorized people to enter the Door of home [8].

In our project, we will use RFID card inductive module KIT which has the following properties:

Based on the MFRC522 Chip Supply Voltage: 3.3V; Current: 13-26mA

Operating frequency: 13.56MHz

Read Range: 0 ~ 60mm (MI fare 1 card)

Interface: SPI, Data Transfer Rate: Max 10Mbit / s Dimensions: 40mm × 60mm

Size of Round Tag: 0.87 × 85.5 × 54 mm Size of Rectangle Tag: 32 x 40.5 x 4.2 mm

Supported card types: mifare1 S50, the mifare1 S70, MI fare Ultralight, MI fare Pro, MI fare Desire

2.4     Design Specifications and Constraints

2.4.1      Design Specifications

Our project contains two parts; the first part is the android application on home owner smart phone to gives the home owner three choices after receiving visitor image. First choice show the image to insure the face of visitor. Second choice record sound message and send it to Raspberry pi. Third choice call police if the visitor is intruder. The second part is the raspberry pi which must be designed with a number of sensor that makes the system able to sense body motion in front of the door, if there is someone in front of the door, the sensor will send a signal to the Raspberry Pi that sends a signal to the camera to take an image. Then it compares that image with family members faces stored in the system by face recognition algorithm if the visitor is identified as family member the system, then requires Radio- frequency identification (RFID) as an additional identification method. Otherwise, it saves the image of that visitor along with the time, then it sends a notification to owner’s smartphone who can send feedback to the visitor, if the owners discover that the visitor is an intruder, he can notify the police by calling them.

The main components of our system.

Raspberry Pi: The main component in the system, which will be linked to the other components. Represent the inlet and outlet of the signals and information.

PIR sensor: Sensing if there is someone in front of the door in range about 3 meters (120- degree cone), then send a signal to Raspberry Pi to give an order to the camera to capture an image.

Face recognition process: should provide real-time and accurate results and check whether the visitor is family member or not

Speaker: used for audio output send by home owner to the visitor.

Pi-Camera: used for images input that capture to the visitors after take an order from Raspberry Pi.

RFID technology: additional identification method at the same purpose as a bar code or ATM card.

Home owner phone: receives images from Raspberry Pi by Wi-Fi and sends feed back to the visitor if need. Call police if there is an intruder.

2.4.2      Design Constraints

• To have quick response to the arrived visitors, we will use the Passive Infrared (PIR) motion sensor to trigger the system functionalities.

• Face recognition algorithm should give real time results, to be sure that the user will be notified in a time when the visitor arrives.

• Face recognition algorithm may give some wrong results, so we need an additional identification method to increase the level of security in our project, so that we will use RFID tags.

System Design

Overview:

The following sections consists of the block diagram of the system, flowchart, and discuss the design options in detail.

3.1     Brief description of the system

The system contains two main parts: the mobile application and the door system which is designed using Raspberry Pi.

Our system will be designed to detects motion in front of the door using PIR motion sensor that sends signal to raspberry pi board. Raspberry Pi sends signal to the pi camera to capture an image, then it sends a notification message to the user’s smart phone, then it compares that image with known faces of the family members by face recognition algorithm and saves them in raspberry pi, if the visitor is identified as a family member the system then waits to read RFID tag to open the door, Otherwise the system will return to sense motion stage.

When the user click on the notification message  at  his  mobile  phone  the  captured image will displayed at image view, the application also provides other options to the user such as: the ability to call police if he felt visitor is an intruder, and the user can record a voice message and send it to the raspberry pi then the speaker connected to the raspberry pi will play the voice for the visitor, the user can also browse all pictures captured by the raspberry pi.

3.2     System Diagrams

In this subsection, two diagrams are represented for extra understanding of the project concepts and design

Block Diagram:

Figure 3.1 contains a block diagram shows the main building blocks of the system and the communications between them.

Hardware Layout Diagram:

This diagram shows hardware components and the connections between them.

Schematic Diagram:

This diagram shows how hardware components connect with each other’s.

3.3     Design options.

1.         The Controller.

We have two options for the controller:

a.       Raspberry-Pi 3. Advantages:

• Can be programmed using a variety of programming languages.
  • Open source advance microcontroller
    • Provide USB and HDMI.
    • Built in Wi-Fi module.
    • Can run OS (Linux OS)
    • Can use with Screen and other peripheral like Keyboard, Mouse.
    • More CPU clock speed some 300 MHz higher than that of the Pi 2.
    • Pi 3 is based around a 64-bit processor, compared to the 32-bit CPU used in the Pi 2.

Disadvantages:

• Need more power than Raspberry-Pi B+.

b.      Raspberry-Pi B+. Advantages:

• Can be programmed using a variety of programming languages.
  • Open source advance microcontroller.
    • Provide USB, HDMI.
    • Can run OS (Linux OS).
    • Can use with Screen and other peripheral like Keyboard, Mouse.

Disadvantages:

• Not built in Wi-Fi module

Finally, we decide to use raspberry-pi 3 since to its advantages and the disadvantages of raspberry-pi B+.

2.   Motion Detection.

We have two options:

a.       Passive Infrared sensor. Advantages:

• It can detect objects from far distances.
  • Small size, inexpensive, low-power, easy to use

Disadvantages:

• Only receive infrared light.

b.      Ultrasonic. Advantages:

• An ultrasonic sensor’s response is not dependent upon the surface color or optical reflectivity of the object.

Disadvantages:

• It has a minimum sensing distance.

Finally, we decide to use Passive Infrared sensor since its advantages and ultrasonic sensor disadvantages.

3.    Identification Method.

1. Face Recognition Algorithm. Advantages:Low cost.The requirement easily founded and compatible of requirement of the project

Disadvantages:

• There is no accuracy in recognizing person’s face.
  • Instability of the system
    • Testing and maintainability of the system is very hard

b.      Fingerprint based system. Advantages:

• More security than face recognition
  • No matches or similar between two people in fingerprint.
    • Reduce time of processing

Disadvantages:

• High cost.
  • Not founded easily.

Finally, we decide to use face recognition algorithm. Because we use the Raspberry Pi 3 that have good computing power and we use RFID tags as an low cost additional identification method to resolve the expected security risk caused by the inaccurate outcomes of face recognition process.

Software

4.1 Brief description of the software

In this chapter, we will describe the software that we have used to build our project.

4.1.1   Smart home application:

Smart home application is the main process in our project that do the majority of its functions, that application monitor the state of the motion sensor when the state of the sensor changed, the smart home application uses the pi camera to take an image and saves it with the date and time when it taken, after that step the application sends notification to the owner’s smart phone ,then it enters the face recognition process it may take additional images if the first image have no faces for recognition ,if the visitor recognized as a family member the application then goes to RFID stage which require the RFID tag then it read it and compares its ID by the authenticated values, when the user enter a valid RFID tag the door will be opened then the system returns to sense motion stage ,if the visitor doesn’t recognized or the entered tag is invalid the system will return to sense motion stage directly.

We use Python programming language to build that part of the system for many reasons first is the simplicity of python programming language syntax and the most important reason is many useful libraries available for python such as OpenCV for image processing and RPi library that gives the access to raspberry pi GPIO and many other libraries that we use to build that application like numpy, sys, math, Image, picamera, time, signal and socket.

4.1.2 Sound Server:

Small application written in java programming language it’s binds at specific port at the Raspberry Pi, the function of that server is to receive voice messages from the android application and play them, we use java programming language here because the android application is built by java and we face some problems when we try to play the voice that recorded by the android application in python server all related to the encoding format.

4.1.3 Image Server:

Small application written in python programming language it’s binds at specific port at the Raspberry Pi, the function of that server is to transfer images from Raspberry pi to the android application, the main purpose to build that part in separate code is to give the user the ability to browse all events without any interception to the system functionalities, and python programming language is used to build that part of the system.

4.1.4 Mobile application:

An essential part of our system it receives the notifications from the Raspberry Pi part and alert the user when the notification arrives, when the user clicks at that notification that application will display the taken image, the application gives the user many features like sending voice messages or calling police and the mobile application can be used to browse the taken images at any time.

We build it using Android Studio IDE and java programming language.

4.2     Software flowcharts

Mobile application flowchart:

System work flowchart:

Validation and Discussion

The following subsections have description of the implementation, implementation challenges, and implementation issues.

5.1    Implementation

5.1.1    Download the operating system on Raspberry Pi

We download a pre-installed copy of Raspbian operating system from raspberrypi.org web site, we installed that disk image in the raspberry pi SD card using Win32DiskImager tool, after insert that SD card in the raspberry pi and connect it to the power supply, then Raspbian operating system was be ready to be used to build our system.

5.1.2    Installing needed packages

Raspbian comes with many useful pre-installed packages such as JDK, python and Others, but we need to install some needed packages for our project the most important one is OpenCV library that will be used to provide face recognitions features to install OpenCV library you need to follow the instructions in: “http://ivanha.github.io/blog/2015/05/16/install-opencv-on-raspberry-pi/”

After that, your OpenCV should be ready to use you can test your installation by:

>>> import cv2

>>> cv2.version 2.4.10

Other packages are SPI-Py and MFRC522-python that runs the MFRC522 module for RFID tags. To install these libraries, you need to run these commands:

git clone https://github.com/lthiery/SPI-Py.git cd SPI-Py

sudo python setup.py install Install MFRC522-python

git clone https://github.com/mxgxw/MFRC522-python.git cd MFRC522-python

5.1.3    Build smart home system

After connecting all parts together, we build smart door system parts that was Briefly descried at the software chapter with flowcharts.

5.2     Implementation challenges

We faced many challenges while implementing our system. But the one that is worth mentioning is the transferring of files between the Raspberry Pi and the android app. It was a very complicated task. This is because we transfer the files between completely different applications and environments.

Many problems and issues have faced due to using different programming languages to design the client and the server. As an example, for that issues when we send sound track from the android application to the python server at raspberry pi we get no more than damaged file. We resolved that problems by split our smart home application to three sub applications the first application is the smart home application that have the whole system functions except sound and image transfer, the second application is the sound server and we writes it in java programming language, the last application is the image server.

5.3     Implementations issues

PIR Sensor and RFID connections:

We connect the sensor and the RFID reader with Raspberry pi GPIO shown in figure 3.2 After connecting raspberry pi with electricity and runs the code, the sensor was not working properly. After extensive research, we found that there are two GPIO modes:

The GPIO.BOARD option specifies that we are referring to the pins by the number of the pin.

The GPIO.BCM option means that we are referring to the pins by the “Broadcom SOC channel” number.

The main problem is that the RFID reader code was written depends on GPIO.BCM mode, and we use the GPIO.BOARD for the PIR sensor connection, with more research we find that we can use GPIO.cleanup method to reset the GPIO mode when moved to RFID reading stage.

5.4     System validation and testing

5.4.1    Face recognitions algorithm validation:

OpenCV library provides three face recognitions algorithms there are Eigen faces, fisher faces and Local Binary Patterns Histograms(LBPH) algorithm, we decide to use the LBPH algorithm for many reasons such as it doesn’t suffer from illumination variations, scale, translation or rotation in images. [9]

At the three cases the recognition function returns the id of the recognized person and threshold for recognition. We try to find the best value of the recognition threshold so the algorithm can identify people and no one will be identified by wrong id after experimental test we choose a threshold of 37, in our experiment we test our application with faces doesn’t stored in the system and the threshold value varies from 45-90, and when we test it on people with stored images the threshold have varied between 15-39, then we calculate the average of known faces we get 37, if we make the threshold less than 30 much of readings will be not recognized and when it sets to be more than 90 all peoples will be recognized, at the 37 the majority of readings are valid.

5.3.2    PIR sensor validation:

PIR sensor also have two potentiometers one of them is used to adjust the delay time, it determines how long the PIR will keep the Output High (ON) after detecting motion. And the other is Sensitivity with this potentiometer you can set the range you have in mind.

We adjust Poth potentiometers to have maximum range and least delay time, that because the sensor doesn’t have stable readings before we do that.

5.3.3    Camera test and connection

We connect the Camera with Raspberry pi CSI as figure 3.2. After connected raspberry pi with electricity and runs the code. The camera capture image as “.jpg” format. we find that we can adjust the resolution of captured image by (cam.resolution =(x,y)) where x and y are the dimensions of the image.

Conclusion and future work Conclusion:

Smart door system sense motion in front of the door it will capture image for the visitor, sends notification message to the home owner who will receive the notification by android application installed in his phone, that application makes the owner able to browse all the captured images in the raspberry pi device and the owner can use it to send voice messages to his visitor or call police, more over the system provides access control feature through the use of face recognitions algorithm to recognize family members and the RFID tags.

Face recognition algorithm determines whether the visitor is family members or not by using OpenCV/python and the RFID to ensure that the visitor is family member through provide unique identifier.

Future Work:

We suggest adding the following feature to the system for any upgrade:

• Make video call between visitor and home owner.
  • Monitoring the home in more than one direction.
  • Integrating the door system into complete home automation system.
  • Use alternative notification method in the case when Wi-Fi connection is unavailable for example by sending SMS messages.

Appendix A : Application screens

Appendix B: System finalization