The goal of our project is to use the Raspberry Pi and create a touchscreen enabled Heads Up Display (HUD) for an automobile. The HUD would display useful, real time information through an easy-to-ready GUI onto a PiTFT screen. Some example information includes the automobile’s speed, acceleration, cabin temperature, and engine RPM etc. We interfaced with a 2007 model Toyota Yaris through its OBDII port using Bluetooth technology and was able to capture real-time engine diagnostics data. We also programmed an Inertial Measurement Unit (IMU) that contained an accelerometer, gyroscope, magnetometer, and temperature sensor in order to capture the vehicle’s directional orientation, acceleration, and cabin temperature.
For this final project, we implemented a Heads Up Display (HUD) using the Raspberry Pi that can interface with any modern vehicle and be able to relay vital information to the driver. We used two main pieces of peripheral hardware, the OBD-II Bluetooth Dongle and a 9-DOF Inertial Measurement Unit (IMU), to capture all the necessary information pertaining to the vehicle. On top of that, we also used a 2.8’’ PiTFT screen to display data to the driver in real-time.
The IMU itself contains four sensors: magnetometer, accelerometer, gyroscope, and temperature sensor. We loaded data from the IMU to the Raspberry Pi using an I2C interface, which required a total of four wires: Vin, GND, SCL, and SDA. Using the IMU, we were able to obtain the car’s acceleration, magnetic direction (compass) and cabin temperature. The OBD-II Bluetooth dongle allowed the Raspberry Pi to acquire data wirelessly from the car’s on board diagnostics computer. Using the OBDII port, the Raspberry Pi was able to obtain vehicle data such as speed, engine RPM, engine coolant temperature, as well as hundreds of others highly specific vehicle related information and error codes. In order to output the obtained information to the driver, we built a touchscreen capable HUD using the pygame module to display a few pre-selected vehicle information onto the TFT screen. The driver was able to toggle certain units such mph or kph based on his or her preference by simply touching on the related parts of the screen.
Design and Testing
Our project was split up into two main parts: data collection from the IMU and data collection from the OBD-II.
The IMU we used was the Adafruit LSM9DS0 breakout board, which contains an accelerometer, magnetometer, gyroscope, and temperature sensor. The 3-axis accelerometer can detect which direction points towards the Earth by measuring gravity or how fast the chip is accelerating in 3D space. The 3-axis magnetometer that can sense from which direction the strongest magnetic force is coming from, generally used to detect magnetic north. Lastly, the 3-axis gyroscope that can measure spin and twist. By combining all this data, we can use the IMU to sense motion, direction, and orientation. In our case, we were able to produce accurate values for the car’s acceleration and the direction to which the car is driving towards. On top of that, the embedded temperature sensor on the breakout board can measure the vehicle cabin temperature, giving us an extra piece of data useful for the driver.
LSM9DS0 is set up such that it can be communicated with using both SPI and I2C interfaces. Due to the relative simplicity and the fewer usage of wires, we opted to use the I2C interface. The Inter-integrated Circuit (I2C) Protocol is a protocol intended to allow multiple “slave” digital integrated circuits to communicate with one or more “master” chips. I2C is only intended for short distance communications within a single device and only requires two signal wires to exchange information. To physically connect the Raspberry Pi to the IMU breakout board, we used a total of 4 wires for 4 pin pairs shown in the table below:
|SDA||I2C Serial Data|
Since the i2c is a communication protocol that runs only over a two wire bus SDA (Serial Data) and SCL (Serial Clock), the same data and clock lines are shared between multiple slaves like the LSM9DS0. We needed to choose which device to communicate with by selecting the appropriate bus address for the IMU. Once we enabled I2C on the Raspberry Pi, we downloaded i2c-tools to help us detect the correct bus addresses for our IMU device. Turns out, the gyroscope on the LSM9DS0 uses the address of 0x6B and the accelerometer, magnetometer, and temperature sensor uses 0x1D.
In order to read the IMU sensor values, we must first write to the register pertaining to the sensor to turn it on and set the sensitivity level. In python, we used the smbus module and specified the device address in order to read and write specific registers on the IMU. For example, in order to turn on the temperature sensor and configure some of its associated settings, we used the function:
#initialize temperature sensor
writeTMP(CTRL_REG5_XM, 0b11110000) #TMP enable, M data rate = 50Hz
The CTRL_REG5_XM refers to the address given to the register that controls the temperature sensor. We wrote an 8 bit value into that register to enable the temperature sensor and set the data rate to be 50Hz. All the information regarding to the register addresses and their corresponding values can be found in the LSM9DS0 data sheet. Once we have enabled the sensor, we can use the read function to output the values stored in the registers. Since each register can only hold 8 bits, we needed to read two registers at once and combined the two values to obtain the actual value. For example, in order to read the temperature value, we used the following function:
tmp_l = bus.read_byte_data(TMP_ADDRESS, OUT_TEMP_L_XM)
tmp_h = bus.read_byte_data(TMP_ADDRESS, OUT_TEMP_H_XM)
tmp_combined = ((tmp_h & 0b00001111) << 8) | tmp_l
tmp_combined = tmp_combined if tmp_combined < 2048 else tmp_combined – 4096
We read both the low byte register and the high byte register, then combine by left shifting the high byte register by 8 bits and ORing both values. Since the final value is in two’s complement form, we had to take into extra consideration in converting the raw binary into integer.
Using the principle above, we can now read all the raw sensor values for the accelerometer, gyroscope, and magnetometer. We physically mounted the IMU onto the the Raspberry Pi so that it can remain stable and all the readings pertaining to the IMU will pertain to the Pi. All in all each sensor produced 3 values for each of the x, y, and z axis, so we had to read from 6 registers for each. The sensors were mostly jittery when read at a high frequency so we implemented kalman filter to predict values based on a series of outputs and historical results. The Kalman filter is an algorithm that uses a series of measurements over time, which could contain noise and other inaccuracies, to produces prediction of the actual value which tends to be more precise than the raw value based on a single measurement.
After obtaining the kalman filtered sensor values, we set to make meaningful sense of the information. In terms of temperature, the output value already made sense and was presented in degrees Celsius. We simply converted that value to Fahrenheit for the predominantly American audience. We used the 3 xyz magnetometer values to calculate a heading value, which after calibration, could accurately output the direction that the Raspberry Pi is pointed towards. We were able to calculate the heading using the following principle found on AN3192 Application Note:
Although our IMU was mounted relatively level to the ground, we added tilt compensation to further increase the accuracy of our heading value due to to the possibility of the vehicle traveling on uneven grounds, such as on a hill. We used the following principle found on the AN3192 application note to calculate the tilt-compensated heading value.
After calculating the tilt-compensated heading value, we built a digital compass on the Raspberry Pi using two images, one containing the background of a compass and another of the compass needle. We used the pygame module to display the compass background, and then rotate the needle in place based on the heading value. After some calibration, we were able to produce very accurate results that was comparable to the iPhone compass app.
In order to calculate the acceleration of the device, we used similar principles for the magnetometer to come up with a tilt-compensated value. Since there is always a 1G of acceleration in the z axis due to gravity, we tried to find the angle at which the device was tilting in order to find accurate x and y acceleration values without gravity. The following excerpt from AN-1057 Application Note on Using Accelerometer For Inclination Sensing details how we calculated the angles at which the IMU was oriented and then use the angles to calculate tilt-compensated acceleration value in the x-y plane.
To test all the sensors, we used the terminal to print out all the values and compared them with sample reference values found in the data sheet. For temperature, it was easy to see whether it was working correctly based on the temperature of the room. For acceleration, we tested by violently moving the device back and forth to see if the acceleration value increased. When the device is level and not moving, only the z axis registered acceleration in the range of 1G. For the magnetometer, we used the digital compass we made and compared it with the compass app from an iPhone to calibrate and check its accuracy. Overall, all the sensors produced sensible values most of the time. Only the temperature sensor gave us issues because it produced erroneous temperature reading on occasions, but did increase in temperature when warmed up and decrease in temperature when cooled down consistently.
The second part of the project involves obtaining the OBD II diagnostics data from the car. On-Board Diagnostics systems (OBD) are used in most commercially available cars and trucks, and use an on board computer and an array of sensors to measure key diagnostic data. These data are used to run the dashboard sensors such as the speedometer and tachometer. With the OBD-II interface, we can request that data and much more, including various error codes. In our project, we poll the OBD-II for data on the car’s speed, engine RPM, engine load, throttle position, air intake temperature and coolant temperature.
We communicate with the car’s diagnostics computer through the OBD-II bluetooth sensor. This is a device that connects directly to the car’s OBD-II port, and facilitates two way communication with the car through bluetooth. The OBD-II sensor and location of the port are shown below.
The above device is powered through the car’s electrical harness, so it needs no batteries or external connections. Once the device is hooked up to the car’s OBD-II port, the car is turned on, and the device is activated. We obtain data from this device by first setting up a connection with it and then sending requests for data. The device, if connected, will respond to these requests by sending a packet over bluetooth containing the data requested.
The next step is to setup the Raspberry Pi to be able to communicate with this device. For this, we first set up a USB bluetooth adapter on the Pi to allow it to pair with nearby bluetooth devices. Once the Pi has access to bluetooth, we pair the Pi with the OBD-II sensor. When the Pi is paired with the sensor, we need a way to have our various programs access this data. To do this, we attach the incoming bluetooth port to a virtual serial port, using the rfcomm tool. Now programs can communicate with the sensor through this serial port.
To setup the OBD-II communications and be able to communicate with the device, we used a library called pyobd, made by Peter Harris. This library abstracts the OBD-II communication protocol, so that we can send and receive data at a higher level, and not worry about the lowest level details such as encryption and decryption. Using pyobd, we can create an obd sensor object, that contains and sets up all the communication with the device. Once the object is created, we send a request to the device to initiate a connection. If a connection is successful, the sensor responds with an ACK and tells the program all the data that it supports. (Not all cars support all the data OBD-II can provide) Once the connection is verified, we send requests for the specific pieces of data we want for the HUD. The sensor responds with a large packet, and we parse that for the sensor data. We then process the data to turn it into human readable numbers with proper units, as the sensor initially gives us a large HEX number with several header bytes.
Once the data is received and processed, the HUD app can simply display it on the screen. The HUD app polls this data at a constant rate, with the speed and RPM data being polled faster than the other data. Below is a visual representation of how the data moves from the car all the way to the HUD display and back:
After obtaining all the necessary vehicle data from both the IMU and the OBDII, we built a simple touch enabled GUI using pygame that would display all the data in an organized format. The driver can toggle the units between international and imperial for certain data such as speed and temperature by touching the location on the TFT screen where the data is displayed. The resulting GUI can be seen at the top of the page.
We began testing the app by displaying only the IMU data, since it didn’t need a connection to the car. To test this we moved the device in all the different previously mentioned measured axes, to see if the acceleration accordingly changed. We then tested the compass by comparing it to the compass on a cell phone. From these tests we noticed that the acceleration was mostly correct when moving in an xy plane since we compensated the value by removing the gravitational acceleration exerted on all the axes on uneven planes. We also noticed that the compass was generally unreliable when used indoors. This is potentially due to all the interference the magnetometer faces in a metallic environment. The temperature sensor worked inconsistently as it would give us erroneous initial temperature readings. However, it did increase in temperature when warmed up and decrease in temperature when cooled down consistently.
Testing the OBD-II data was slightly more difficult, as it requires a constant connection to the car. After trying various setups, we realized that the easiest way to debug and develop the OBD-II code was to have the Pi inside the house, connected to the house wifi and via bluetooth to the OBD-II sensor. The car was parked and running just outside the house, close enough to get a bluetooth and wifi connection to the car. To develop the code, we went into the car with our laptops, and SSHed into the Pi. At first we had difficulties setting up the bluetooth connection, as the sensor will only pair with one device upon turning on. Also, if you reboot the Pi, the sensor sees it as a new session, and will not automatically pair up with the Pi again. This means you have to restart the car to restart the sensor, until the Pi and car have been restarted and the two devices can pair up properly. We then also found that there needs to be specific actions taken to connect the bluetooth port to the virtual serial port. After experimenting and researching for a while on the rfcomm utilities, we were able to get a consistent connection to the serial port.
In testing the OBD II code, the biggest problem was unhandled errors in the pyobd library. With the library as given, there were numerous basic errors that would crash the whole application. Upon inspection, these were simple errors such as the OBD II not handling a certain code or not responding with the exact string that was expected. (My car would prepend a “#” to every sensor message, and this was unhandled in the pyobd library) To fix this, we went into the pyobd files, tracing all the errors and creating new ways to handle them, or new ways to parse the message if it differed slightly from what was expected. Once we had data consistently coming out of the sensor, we had to make it human readable. The library was supposed to handle this, but when we ran tests, we got nonsensical values, such as an RPM of 1364567800 or a speed of -53492.23897. Looking into this, we realized that the data comes in with several header bytes, that are meant to be stripped off. They weren’t however, so the code was considering all those bytes as part of the sensor data. We then went into the code and did our own data conversions to strip off the header bytes and properly convert the data into human readable units.
We first tested the OBD-II data by running diagnostics on my car idling in the parking lot. We verified that the speed was 0MPH, and the RPM idled around 700. We tested revving up the engine a bit, verifying the increase in RPM. We verified the coolant temperature by judging it to be in a reasonable range (175F) and double checking it with an OBD-II Android app. In a similar fashion, we verified the engine load, intake temperature, and throttle sensor data were reasonable. This was the extent of our testing ability in the idle car, so to test our HUD further we took to the streets. This involved removing our live computer connection to the Pi, as we’d leave the WiFi range. We started in the parking lot, powering the Pi off a car cellphone charger, connected to the house WiFi. We started up the HUD app via WiFi and then drove off. In our driving tests, we verified that the speed would move up and down, matching closely with the dashboard speedometer. We also verified that the IMU data was working in the moving car environment. We also further tested the RPM by having a wider range of testpoints than the idling car. To test various situations, we started out driving flat, then downhill, then uphill. We noticed the engine load and throttle percentages rising on the uphill segments, as expected. All the data were fairly responsive, although there is a slight lag, especially in the OBD-II obtained data. We have determined this to be an effect of the bluetooth connection, which can only be so fast. To test this theory, we ran the Android OBD-II app as a benchmark, which had a similar amount of lag.
In conclusion, our project is a reliable, accurate HUD that can be used for any car that has OBD-II capabilities. The biggest issues we had were in bluetooth, but once that was flushed out we started getting very reliable information from the OBD-II and IMU. The biggest lingering issues are in the IMU thermometer and the bluetooth lag. The IMU thermometer was expected to be fairly unreliable, as it’s located very close to the Pi and is a surface mount chip. We assume that the chip was damaged when we soldered on the pin headers. The compass is also often buggy, but this error is minimized when in the car, outside, away from the interference found in a lab or living room. The lag from the bluetooth is present, but isn’t bad enough to cause any problems, and it’s slight enough to not be a distraction. To fix this we’ve considered hardwiring the OBD-II sensor to the Pi’s GPIO pins, eliminating the need for bluetooth. This HUD offers digitized versions of information available already on the car’s dashboard, as well as new information from the OBD-II and the IMU, giving the driver an enhanced awareness on the road. This biggest advantage of this project is that the information that was previously obscured inside the car is now available to be used on the Pi, opening up limitless possibilities for future developments.
The OBD-II connection really lends itself to a lot of potential expansions in our app. We only used 6 sensors in our HUD, but there are many more that we could potentially add, and we might have the user choose which sensors to display on which parts of the screen. The OBD-II also reads out error codes for the car. We were able to use the OBD code to obtain an error code via command line, but we never integrated this with our HUD. In a future rendition of our app, we might have a tab that allows the user to see the error codes present, along with what they mean and what it means to resolve the error. Using the collected data, we could also determine the fuel efficiency for the vehicle. With this data, we could determine mileage for certain regimes of the car’s operation, and use this data to estimate the current mileage and current miles left in the gas tank. We might also collect the data we display over time into a database, so the driver can view the car’s performance and the driver’s habits over time. As far as the IMU, we couldn’t find a way of integrating the gyroscope data, so that would be on the list of potential improvements. Another improvement is to detect the difference between acceleration and deceleration. Right now, the captured acceleration values returns positive regardless whether the driver is pedaling or hitting the brakes. In the future we could use the orientation of the IMU to detect which way the vehicle is moving and calculate acceleration and deceleration values accordingly.
B. Source Code
- Car_HUD.py (The main file for Car HUD)
- HUD.py (Demo program to display HUD without OBD2)
- LSM9DS0 (Header file for IMU)
- temp_read.py (Program that reads temperature)
- rpiIMU.py (Program that calculates acceleration and heading)
- pynog.py (Program that initiates OBD2 and captures data)
- kalman_filter.py (Kalman filter library)
- obd_io.py (Helper library for OBD2 communication)
- obd_sensors.py (Library to convert OBD2 data)
- obd2_utils.py (Library for various OBD2 utilities)
- obd2_codes.py (Header file holding all OBD2 error codes)
- I2C Tutorial
- IMU and Compass Tutorial
- LSM9DS0 Hookup Guide
- LSM9DS0 Datasheet
- Tilt Compensation Calculation (AN-3192 Application Note)
- Orientation Angles Calculation (AN-1057 Application Note)
We would like to thank Professor Skovira for teaching such a great class on the Raspberry Pi and giving us the chance to work on a project like this. We would also like to thank all of the TA’s for the class for helping us in the past.
We would also like to thank Paul Swirhun and Shao-Yu Liang, whose website template helped this site to look much better than we could have done without them
Also, special thanks to Will’s Toyota Yaris for finally cooperating with us and Luna’s for late night motivational pizza.