Eagleye: Augmented Reality Personnel Tracking Integration

Introduction

Problem Statement

There are inherently a lot of problems with not knowing where people are located at. Whether
they are in the location they say they are, or if they are missing and others are trying to find them.
When workers cannot be located, there are safety concerns, for example if you are working with
something dangerous that could harm somebody such as demolition on a building. It can also
mean people can get hurt doing something they should not be doing, and then lie because you
have no way of seeing where they were and what they were doing. Specifically for this project,
Optical Operations LLC wants to solve the problem of general contractors not having access to
their personnels’ locations for construction megaprojects (projects costing upwards of $1 billion).
Not having accountability for workers leads to loss of money and time on these projects and can
make a huge difference in budget and schedule. This problem boils down to both a safety issue
and a productivity issue.

Project Statement

This project is meant to ultimately create an MVP or minimum viable product that our client can
show to investors to gain capital for his ventures. This project involves three distinct parts:
tracking, the HoloLens mixed reality application, and the microservices that serve as the
“middleman” between the tracking token and the HoloLens application. We divided our team
into three separate subteams accordingly, having two members work on each team. This
divide-and-conquer method allowed us to maximize the time we had this semester to accomplish
the goals stated below.

Project Goals

The goal of the project was to create a Microsoft HoloLens application that shows a live map of
where different people are at any given time. The personnel are tracked through a small token,
which is attached to the workers Personal Protective Equipment (PPE). This small token
communicates with a distributed tracking system that relays data to the HoloLens, and a
supervisor will utilize this HoloLens application for monitoring purposes.

Intended Uses and Users

The primary user for this system is a supervisor on a construction site. They will be using the
HoloLens application to view the workers on their site. The secondary users will be the
construction workers who are wearing the token on their belt clips. The comfort and satisfaction
of both types of users are important to us because we want to make sure the token does not

interfere with the equipment they are required to wear or with their daily tasks. Since the
companies will be investing a significant amount of money in the HoloLens to use the product, we
want to make sure they are investing in a product that is useful for them and provides a pleasant
experience for the user.
The primary use of our system is to allow a construction supervisor to visualize where their
workers are at a given time and to be able to play back a visual to see data within the day. This use
represents the core functionality of the product we will be building.
The aforementioned use is primarily for productivity purposes. Having real time locations and
saved daily locations of all personnel on a megaproject can hugely help increase productivity. It is
estimated that every day that a mega project goes over the allotted time scheduled costs an
average of 3.3 million dollars. Therefore, increasing productivity can save construction companies
massive amounts of money.

Additionally, the HoloLens application will be used for safety purposes. One in five worker deaths
in 2015 occurred in a construction work environment [1]. These numbers could be decreased if
supervisors were able to know where their employees were to avoid dangerous situations.
Expected End Product and Deliverables (Established During First
Semester of Work)
● The end product shall have a tracking service.
● The end product shall be a scalable outdoor personnel device.
● The end product shall have documented source code.
● The end product shall be a Proof of Concept for the client to showcase to investors.
● The end product shall have an UI for tracking for Microsoft HoloLens.
● The end product shall have some possible casing for the product.
● The end product shall have reliable communication between devices.
● The end product shall be demonstrable to future customers and investors.

Design

Specifications

The main focus of the project is to get a system in place to track multiple personnel and display
their locations on a virtual map. The initial product should be able track up 6 people on the
worksite in real time with a same delay. Another design specification is the accuracy of our
tracking. We would like to make our solution as accurate as possible, with a minimum accuracy of
less than 5m in error.
In addition, the tracking token that will be placed on each personnel must last at least an average
work day (10 hours). It also has to be able to relay readable data to the HoloLens to be
displayed.This token has to be able to withstand rough conditions, must be water resistant, and
reasonably sized so it will not hinder the worker wearing it. We would like it to attach to PPE so it
must always be worn. The relay will also be secure so unauthorized users cannot access the
information or create false data. The final requirement is that the system must be scalable, so it
will be able to handle more people at larger construction sites, as well as maintainable to last the
length of the construction project.

Functional Requirements:

● 3-part system: token, services, and HoloLens
● System will utilize construction sites’ wifi
● Worker will keep token on belt and it will use wireless signal strength data for localization
● Admin website
● Must store data for future playback

Non-Functional Requirements

● Must track at least 6 people in a playground-sized environment (20 x 20 m): the end goal
is to be able to track our group successfully in a small, coned-off simulated outside work
environment.
● Must be accurate within 5 meters: the tracking must show the avatar as accurately as
possible to track people in real time.
● Token relays information in a readable format to HoloLens.
● If active sensor, battery life = 1 work day (10 hours): needs to be able to last an entire
workday without being charged.
● Sensor communication range: 10m: must have a minimum range of 10 m to limit the
number of sensors needed.
● Real time tracking: acceptable delay of 1 second: if there is too long of a delay, the
positioning will not be accurate on a moving target.
● Track people moving at maximum of 5 mph: this will allow the positioning to still be
accurate on someone moving at a reasonable speed.

System Design

The design that our team has chosen is using RSSI (Received Signal Strength Indicator) data and
hardware tokens to track the personnel, and then send that information to our server and display
it for the user. The team also chose to have a base station that will increase the accuracy of the
system. The job site will be completely covered in WiFi with an appropriate amount of access
points to guarantee that anywhere the personnel go, they will be in the range of at least three
access points.
The token will be a Raspberry Pi Zero, which will be attached to the toolbelt of the worker. This
token will gather the RSSI data, and send it through the network to a hub that uses a path loss
function to convert the RSSI data into an accurate location for the multiple tokens. Once it
generates the location of the tokens, it will send this data to an offsite server. From this server, we
will get the information and generate avatars at the locations on a 3D mapping of the site on the
HoloLens.
The base station is also a Raspberry Pi Zero that gathers RSSI data from multiple fixed locations
and will find the optimal TxPower and Gamma Value for the Path Loss Function. It will then send
this data over to our server to use those values when calculating distance.

L is the path loss in decibels, n is the path loss exponent, d is the distance between the transmitter
and the receiver, usually measured in meters, and C is a constant which accounts for system losses
In order for a customer to set up a worksite, they will be able to use the Admin Website that we
have created to place access points based on latitude, longitude and altitude. It also allows admins
to create users for work sites.

High Level Overview

Overall System

Personnel Tracking Dataflow

Design Analysis
We have compiled the areas of weakness and strength of this design:
● Areas of Weakness
○ Accuracy: The accuracy of our tracking with RSSI will not be as strong as with
another technology such as CSI (Channel State Information). It will likely be
within 5 m, whereas CSI could have provided us with less than 1 m accuracy.
○ IP Addresses: Due to working in a Cloud Environment that is not ours, the client
does not have a Static IP Address if they choose to go from the existing Cloud
Infrastructure (GCP) to another (AWS, Azure)
○ Issue with scaling the Socket-Service as it always has to hold the websocket it is
currently connected to. No way of serializing a web-socket.
● Areas of Strength
○ We are able to use any programming language for the backend services of the
system.
○ With RSSI, every WiFi-connected device can be tracked and see the data easier.
○ It can be used with a closed wifi network
○ The solution with RSSI that we have designed is scalable for several worksites.
○ We are secure with HTTPS routing, JSON Web Tokens for authentication and
limited access to our database
○ With Kubernetes for our container management system, we are easily able to
switch from any Cloud Environment (AWS, Azure, GCP, IBM) and allows for auto
scaling so our client does not have to worry about how to deal with an increase in
load.
○ Our design is a platform that can be added on to in the future. We have created an
API to easily get data from Google BigTable.
○ Since we are using the Microsoft HoloLens over HTC Vive or Oculus Rift, the user
will be using a mixed reality device, which is less draining and uncomfortable than
a virtual reality device. The Microsoft HoloLens allows users to still interact with
their environment as well.

Implementation Details

Hardware and Software

● Hardware
○ Raspberry Pi Zero W (Token/Base Station)
■ Attached to users PPE
○ Cisco Meraki MR74 Access Points
■ Act as reference points for multilateration
○ Microsoft HoloLens
■ Worn by construction supervisor in indoor setting
● Software
○ Tracking
■ Connect to Services via WiFi
■ Scan for surrounding wireless access points
■ Report findings
○ Services
■ Triangulation
● RAM redis server
● Algorithms
● Scheduling Algorithm
■ Save User Location
● Database Wrapper
■ Location Tracking Data (LTD)
● Socket Connection
● RAM redis server
■ Setup
● Backend
● Frontend
○ Microsoft HoloLens
■ Frontend functionality
● Menu display
● Display Map
■ Backend functionality
● Communicate with LTD
● Push information to Frontend

Interface Specifications

Hardware

We used Raspberry Pi Zeros as a main hardware tool for tracking due to their flexibility
and low cost. In order to keep the form factor of the tracking device small, we use the
microcontroller’s onboard Wifi chipset. We will be gathering RSSI on the Raspberry Pis using the
linux built-in tool for scanning wireless networks, iwlist. Our client has bought 6 Raspberry Pi
Zeros for us to use for our demo. For displaying the 3D map of the the worksite the HoloLens, the
only additional configuration required for the HoloLens is network connectivity to interface with
the HoloLens service, so it can get and display personnel location on its map. The Access Points
that our client plans on using for the purpose of known locations in our tracking system is Cisco’s
Meraki APs. Currently we have 5 APs setup in the parking lot of the ISU Startup Factory.

Software

In terms of software, our team uses a multitude of different frameworks, languages, and
platforms, including: Unity, Raspbian, C#, Python, Git, Nodejs, Kubernetes, Google Big Table, and
NGINX. Unity and Visual Studio are needed to run, build, and test the Microsoft HoloLens
application. The code for the HoloLens is first built and tested in Unity and then using the Unity
built-in tools for building an application for various platforms, it then must be built for UWP.
Once built for the HoloLens, it can be sent to the HoloLens or HoloLens Emulator via Microsoft
Visual Studio. Raspbian lite is the base OS for Raspberry Pis. We use Raspbian lite because it does
not have a desktop environment, for the purpose of saving resources so that the device can last
longer on battery power. The program that runs on the Pis relies on python and packages for
sending http request to our web services. For the various backend services we run to support the
the tracking system the technologies we use varies greatly. Each publicly facing endpoint uses an
NGINX server to process requests and implement a secure HTTPS connection to the connected
party. Taking the path of a tracker/base station request, the first services they will hit after the
NGINX server will be Base Correction and Data Entry Services. These services then talk to Redis
and the multilateration estimator. Other software routes on the services side are detailed above in
the overall system diagram.

Testing Process and Results

Testing Process

● Token relays information in a readable format to HoloLens.
○ We tested this by continually changing data format to ensure that the HoloLens
was retrieving the data correctly.

● Realistically-sized token: wearable device attached to belt that is no bigger than an iPhone
8
○ Our test strategy was to test multiple devices to find one that felt the most natural

● Create a system that is scalable to megaprojects (projects greater than one billion dollars):
must be able to scale for projects that have a lot more than six workers.
○ We tested this by attempting to connect to services with as many people as
possible to get an idea of how many people can initially connect without it
breaking
○ Also we attempted to design code for design clarity and modifiability to make sure
when the number of necessary users change, our code can be changed to fit needs
● Maintainable for length of project: for construction site, must be able to be easily
maintained for length of construction project for at least five months at a time.
○ Tested by continuously testing the project as time goes on to ensure that it is
maintained
● If active sensor, battery life = one work day (10 hours): needs to be able to last an entire
workday without being charged.
○ We tested this by leaving tester on for length of a work day with program running
● Must be accurate within five meters: the tracking must show the avatar as accurately as
possible to track people in real time.
○ We tested this by having people carry the tokens in known locations, allowing us
to compare the location we detect from the token and the location we know.
● Must track at least six people in a playground-sized environment (20 x 20 m): the end goal
is to be able to track our group successfully in a small, coned-off simulated outside work
environment.
○ We tested this by having six or more users wear the hardware token (Raspberry Pi
Zero devices) in an area that was 40 x 40 m in size, and monitor the tracking data
to ensure it fits within our accuracy.

Testing Results

● Must track at least six people in a playground-sized environment (20 x 20 m): the end goal
is to be able to track our group successfully in a small, coned-off simulated outside work
environment.
○ We ended up changing our testing strategy to enhance our accuracy, so we now
have less people with multiple tokens receiving data.
● Must be accurate within five meters: the tracking must show the avatar as accurately as
possible to track people in real time.
○ From the testing, we ended with an average accuracy just outside five meters
accuracy, hovering around six meters.
● Token relays information in a readable format to HoloLens.
○ The test results for this were very positive, as by the end of the project we were
able to successfully relay all important information from the token, through the
services, and to the HoloLens.
● If active sensor, battery life = one work day (10 hours): needs to be able to last an entire
workday without being charged.
○ Using portable battery packs, we were able to get well over the 10 hour goal we set
for ourselves

● Sensor communication range: 10 meters: must have a minimum range of 10 meters to limit
the number of sensors needed.
○ Walked around demo area and confirmed that anywhere inside the area could
access information from at least three access points.
● Real time tracking: acceptable delay of one second: if there is too long of a delay, the
positioning will not be accurate on a moving target.
○ Our results for this goal were harder to determine, as we ended up having to
change our connection from sockets to http request, slowing down the data
transfer. This led to a longer delay in the tracking.
● Maintainable for length of project: for construction site, must be able to be easily
maintained for length of construction project for at least five months at a time.
○ Have been successfully running the tests throughout the whole semester, which
has been approximately the amount of time we wanted to prove.

Related Products

Construction
● Buildertrend – The #1 Construction Management Software for home builders & remodelers
● B2W Track – Field Tracking & Analysis
● Accuware – Indoor and GPS location tracking components(https://www.accuware.com)
● QTRX systems – Locate, Map, Track. Indoors (https://www.trxsystems.com).
Augmented Reality Construction
DAQRI Worksense – Professional Augmented Reality. DAQRI designed a suite of AR features for
professionals. The Worksense platform allows clients to use Augmented Reality in the workspace
by providing communication tools, 3D scanning, tracking tags, real-world models, and AR
guiding. (https://daqri.com/worksense/)

Overview

Many related products are associated to individual aspects of our product, but our project does
not have any competitor that provides our exact solution. As a result, many of the related
products are not directly competing with our client. The products discussed earlier are products
that relate to software products that have intentions to sell to construction companies. DAQR’s
product is closer to our solution, but does not seem to have the intention to sell a product that
tracks construction workers or vehicles.

Source: Eagleye: Integration of Personnel Tracking in an Augmented Reality Environment

Eagleye: Integration of Personnel Tracking in an Augmented Reality Environment