Using EEG data to train a TensorFlow model on relaxed mind-states; the beginning of moving a mind to a machine!
What and Why
I’ve always been fascinating by the idea of moving ones mind to a robot to live forever as a machine so I thought what better time to start the seeds of this with a nice bit of Raspberry Pi work with TensorFlow and a Muse brainwave reading Headset – starting off simple and working my way up:
Key Goals:
- Read brainwave data.
- Use TensorFlow to train a model on brainwave data to determine a relaxed or a non-relaxed mindset.
- Use information gathered from this as a jumping off point to see what else can be read from the mind.
I decided on the name Mind Patterning as in the video game Total Annihilation (the thumbnail for this post is artwork from the game) the story goes that humanity found a way to transfer human minds into robots and the process is called patterning. I am also inspired by Dr Samuel Hayden from DOOM 2016 – who transferred his mind into a super combat chassis.
Stuff:
How
First I needed a way to get mind data after some Googling I found a head band exists called the Muse – this can read EEG data from the wearers mind and be used with an app to determine how relaxed someone is during meditation sessions.
With some further reading I realised it could be hooked up to a Raspberry Pi with Bluetooth rather easily and that it could then spit out the raw sensor data into a CSV file. Looking into the code here I discovered that there is code to turn the raw sensor data into more usable brainwave data and then write that to a CSV.
There was some extra stuff I needed to do before the headset would work, I noted the sites I used for instructions:
- https://blog.adafruit.com/2018/05/28/recording-brainwaves-with-a-raspberry-pi/https://github.com/alexandrebarachant/muse-lsl
- https://stackoverflow.com/questions/33684894/numpy-disutils-system-info-notfounderror-no-lapack-blas-resources-found
- https://stackoverflow.com/questions/29586487/still-cant-install-scipy-due-to-missing-fortran-compiler-after-brew-install-gcc
I then went to here to learn about loading CSV data into a TensorFlow model and training on it.
You can find out about installing TensorFlow on a Pi here.
The initial testing of the model training and inference was done with the PipelineRunnerTest.py file which will run through and create an artificial set of training data and then again for testing data, it then calls the CSV pre-processing which then loads and trains the model, then it runs that against the test data.
This allowed me to see easily and quickly how well the model would even work against similar numbers to what the real world data was. It’s also very useful for quickly making adjustments to the training in terms of epochs/batch sizes and what optimiser to use etc.
The CSV data looks like this (example of artificially generated data):
In the end I stuck with very similar settings to what the CSV loading tutorial taught me as that seemed to work pretty well but I will be experimenting to see if it can be made more efficient, by adding layers etc.
The actual data I will be gathering and using will be Alpha, Beta and Theta waves calculated from the raw data from the EEG sensors on the headband.
For more information about these waves have a look here, as a high level summary – essentially the differing of measurements from these 3 waves are the result of the mind being in certain states; for instance a certain wave being higher and another being lower means a relaxed state of mind and vice versa. So this data can be used to detect the state of mind and train the model I want to make in TensorFlow.
At this point I’ll mention my third inspiration for this project – Ira Graves from the Star Trek TNG episode ‘The Schizoid Man‘, where he transfers his mind into Lt. Data to avoid death.
You can get the code from my GitHub.
So lets walk through the modules of code (using Python 3):
TestDataGeneratorCSVbatch:
Generates an artificial data set, can be used to determine the number of lines written, the type of data being written and also the output file name of the data.
This is modified from the code here.
CSVCombiner:
Takes a number of files with certain filename templates (such as beginning with ‘input_test_data_’) and combines them into a single CSV with a set name. This is where the multiple instances of gathering data are put together into a single CSV for the CSVDataImporter. Thanks to the code from here.
CSVDataImporter:
This is where the CSV data is imported and numerically packed into variables that TensorFlow can handle in the below module. This is code from here.
BrainWaveTrainer:
Takes data from the data importer and trains a model on it and then outputs that model to data inference below. This is code from here.
BrainWaveDataInference:
This runs the test data against the model that was built above and then displays the results (a sample of 50). This is code from here.
RealDataGatherer.py
This gathers relax data and then non-relaxed data from the EEG headband, instructing the user wearing it to relax/non-relax their mind. Then calls the RealBrainWaveData module which gathers the data and writes to separate CSV files for each run, then finally calling the combiner module to put them into one big CSV.
It’s done in a way that per-call it will request for a batch of relaxed mind data (user relaxes, clears mind) then asks for non-relaxed (user does best to think and fill mind with as many thoughts and be as non-relaxed as possible).
I did this in a way that gathers smaller batches of mind data and then combines them int a bigger set, this is because sustaining a relaxed mind for long is difficult for me, so it’s easy to batch it up with small samples of when I get my brain to just be quiet.
RealBrainWaveDataCSV.py
This is the module that grabs the data from the EEG module and puts it into CSV files, calling the stream module as a seperate thread (because if the muselsl stream is called it takes over the console and the code won’t move forward until the stream has finished). Then it takes in a number of seconds of data, that can be passed in – then calculates the sensor data into Alpha/Beta/Theta waves that get written to a CSV.
This first recorded 30 seconds of EEG data at a time, then after some testing it was raised to 2 minutes – although I will probably make this configurable at some point.
This is modified from the code here.
MuseStreamBegin.py
This code is from here it basically just starts a connection with the Muse headset and begins a data stream.
PipelineRunnerTest.py
This runs the full pipeline to test the training and inference using artificial data:
- Generates CSV data – train/test
- Imports and Pre-processes data
- Trains model on data
- Inferences and shows results
PipelineRunnerReal.py
This runs the full pipeline but with real data instead of artificial data:
- Gathers CSV data from headset – train/test
- Imports and pre-processes data
- Trains model on data
- Inferences and shows results
Both pipelines use a batch size of 250, epochs 10 and the test runner creates a total CSV size of around 688, 891 rows for train and 68, 281 for test by default, but these can be adjusted with args passed into the pipeline (-h at the end of the files to get the info on args, for example; ‘python3 PipelineRunnerTest.py -h’).
colors.py
This is used to color the print output when running a pipeline.
This is from here.
utils.py
This is some extra gear that game down with the muse-lsl stuff from above.
Each of these can be called independently also to test their individual features, much like a unit test.
The pipeline runners are the main ones though, allowing me to test the entire pipeline with artificial data and then also with real data. Let’s see what the results are like…
Testing & Results
PipelineRunnerTest
With the current configuration of train size, batch size and epochs I managed to get consistently around these results with the artificial data:
Test Loss 0.27878854962912475, Test Accuracy 0.8545414805412292 Done!Running Predictions on Model…2020-02-21 20:57:38.879578: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Out of range: End of sequence[[{{node IteratorGetNext}}]]Predicted relaxation level: 100.00% | Actual outcome: RELAXEDPredicted relaxation level: 46.33% | Actual outcome: RELAXEDPredicted relaxation level: 26.65% | Actual outcome: RELAXEDPredicted relaxation level: 46.43% | Actual outcome: NOT RELAXEDPredicted relaxation level: 0.00% | Actual outcome: NOT RELAXEDPredicted relaxation level: 100.00% | Actual outcome: RELAXEDPredicted relaxation level: 92.04% | Actual outcome: RELAXEDPredicted relaxation level: 8.23% | Actual outcome: NOT RELAXEDPredicted relaxation level: 46.43% | Actual outcome: RELAXEDPredicted relaxation level: 100.00% | Actual outcome: RELAXEDPredicted relaxation level: 0.01% | Actual outcome: NOT RELAXEDPredicted relaxation level: 91.67% | Actual outcome: RELAXEDPredicted relaxation level: 0.00% | Actual outcome: NOT RELAXEDPredicted relaxation level: 100.00% | Actual outcome: RELAXEDPredicted relaxation level: 0.00% | Actual outcome: NOT RELAXEDPredicted relaxation level: 100.00% | Actual outcome: RELAXEDPredicted relaxation level: 0.00% | Actual outcome: NOT RELAXEDPredicted relaxation level: 50.64% | Actual outcome: RELAXEDPredicted relaxation level: 44.01% | Actual outcome: NOT RELAXEDPredicted relaxation level: 6.16% | Actual outcome: NOT RELAXEDPredicted relaxation level: 100.00% | Actual outcome: RELAXEDPredicted relaxation level: 1.28% | Actual outcome: NOT RELAXEDPredicted relaxation level: 100.00% | Actual outcome: RELAXEDPredicted relaxation level: 0.00% | Actual outcome: NOT RELAXEDPredicted relaxation level: 46.23% | Actual outcome: NOT RELAXEDPredicted relaxation level: 8.29% | Actual outcome: NOT RELAXEDPredicted relaxation level: 0.00% | Actual outcome: NOT RELAXEDPredicted relaxation level: 100.00% | Actual outcome: RELAXEDPredicted relaxation level: 50.51% | Actual outcome: RELAXEDPredicted relaxation level: 0.00% | Actual outcome: NOT RELAXEDPredicted relaxation level: 6.24% | Actual outcome: NOT RELAXEDPredicted relaxation level: 6.21% | Actual outcome: RELAXEDPredicted relaxation level: 46.43% | Actual outcome: NOT RELAXEDPredicted relaxation level: 81.17% | Actual outcome: RELAXEDPredicted relaxation level: 100.00% | Actual outcome: RELAXEDPredicted relaxation level: 6.00% | Actual outcome: NOT RELAXEDPredicted relaxation level: 91.34% | Actual outcome: RELAXEDPredicted relaxation level: 46.63% | Actual outcome: RELAXEDPredicted relaxation level: 46.24% | Actual outcome: NOT RELAXEDPredicted relaxation level: 0.00% | Actual outcome: NOT RELAXEDPredicted relaxation level: 0.00% | Actual outcome: NOT RELAXEDPredicted relaxation level: 46.58% | Actual outcome: NOT RELAXEDPredicted relaxation level: 50.51% | Actual outcome: NOT RELAXEDPredicted relaxation level: 0.00% | Actual outcome: NOT RELAXEDPredicted relaxation level: 91.53% | Actual outcome: RELAXEDPredicted relaxation level: 92.12% | Actual outcome: NOT RELAXEDPredicted relaxation level: 46.34% | Actual outcome: RELAXEDPredicted relaxation level: 13.53% | Actual outcome: NOT RELAXEDPredicted relaxation level: 90.99% | Actual outcome: RELAXEDPredicted relaxation level: 46.67% | Actual outcome: RELAXED
Totalling 40/50 generally on the right track with the predictions.
So that’s pretty reasonable, I need to tweak the settings at some point a bit further, but for now these are fairly decent results that are recreatable and reasonably consistent with the test-runner.
PipelineRunnerReal
First results from real EEG set:
Test Loss 0.729458287358284, Test Accuracy 0.9105960130691528 Done!Running Predictions on Model…2020-02-18 21:59:41.875691: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Out of range: End of sequence[[{{node IteratorGetNext}}]]Predicted relaxation level: 43.77% | Actual outcome: NOT RELAXEDPredicted relaxation level: 74.09% | Actual outcome: RELAXEDPredicted relaxation level: 25.71% | Actual outcome: NOT RELAXEDPredicted relaxation level: 0.05% | Actual outcome: NOT RELAXEDPredicted relaxation level: 78.86% | Actual outcome: RELAXEDPredicted relaxation level: 34.37% | Actual outcome: NOT RELAXEDPredicted relaxation level: 71.40% | Actual outcome: RELAXEDPredicted relaxation level: 15.63% | Actual outcome: NOT RELAXEDPredicted relaxation level: 35.14% | Actual outcome: NOT RELAXEDPredicted relaxation level: 92.93% | Actual outcome: NOT RELAXEDPredicted relaxation level: 37.49% | Actual outcome: NOT RELAXEDPredicted relaxation level: 29.67% | Actual outcome: NOT RELAXEDPredicted relaxation level: 44.10% | Actual outcome: NOT RELAXEDPredicted relaxation level: 92.48% | Actual outcome: RELAXEDPredicted relaxation level: 73.76% | Actual outcome: RELAXEDPredicted relaxation level: 40.67% | Actual outcome: NOT RELAXEDPredicted relaxation level: 5.46% | Actual outcome: NOT RELAXEDPredicted relaxation level: 77.27% | Actual outcome: RELAXEDPredicted relaxation level: 84.70% | Actual outcome: NOT RELAXEDPredicted relaxation level: 0.00% | Actual outcome: NOT RELAXEDPredicted relaxation level: 89.17% | Actual outcome: RELAXEDPredicted relaxation level: 77.66% | Actual outcome: RELAXEDPredicted relaxation level: 81.35% | Actual outcome: RELAXEDPredicted relaxation level: 76.97% | Actual outcome: RELAXEDPredicted relaxation level: 43.19% | Actual outcome: NOT RELAXEDPredicted relaxation level: 70.66% | Actual outcome: RELAXEDPredicted relaxation level: 79.35% | Actual outcome: RELAXEDPredicted relaxation level: 34.59% | Actual outcome: NOT RELAXEDPredicted relaxation level: 18.56% | Actual outcome: NOT RELAXEDPredicted relaxation level: 41.35% | Actual outcome: NOT RELAXEDPredicted relaxation level: 99.99% | Actual outcome: RELAXEDPredicted relaxation level: 93.98% | Actual outcome: RELAXEDPredicted relaxation level: 77.95% | Actual outcome: RELAXEDPredicted relaxation level: 77.76% | Actual outcome: RELAXEDPredicted relaxation level: 31.27% | Actual outcome: NOT RELAXEDPredicted relaxation level: 78.86% | Actual outcome: NOT RELAXEDPredicted relaxation level: 79.47% | Actual outcome: RELAXEDPredicted relaxation level: 93.80% | Actual outcome: RELAXEDPredicted relaxation level: 0.03% | Actual outcome: NOT RELAXEDPredicted relaxation level: 3.85% | Actual outcome: NOT RELAXEDPredicted relaxation level: 32.03% | Actual outcome: NOT RELAXEDPredicted relaxation level: 84.88% | Actual outcome: RELAXEDPredicted relaxation level: 82.69% | Actual outcome: RELAXEDPredicted relaxation level: 19.66% | Actual outcome: NOT RELAXEDPredicted relaxation level: 8.53% | Actual outcome: NOT RELAXEDPredicted relaxation level: 81.61% | Actual outcome: RELAXEDPredicted relaxation level: 6.60% | Actual outcome: NOT RELAXEDPredicted relaxation level: 83.50% | Actual outcome: RELAXEDPredicted relaxation level: 41.63% | Actual outcome: NOT RELAXEDPredicted relaxation level: 76.82% | Actual outcome: RELAXED
Totalling 47/50 generally on the money with the predictions. Another attempt yielded similar results.
I tried some more tests and found that the 30 seconds per CSV batch of mind data wasn’t always accurate so I decided to raised it to 2 minutes and a run from that resulted in:
Test Loss 0.434320068359375, Test Accuracy 0.7653399705886841 Done!Running Predictions on Model…2020-02-21 22:08:07.551755: W tensorflow/core/common_runtime/base_collective_executor.cc:217] Bas eCollectiveExecutor::StartAbort Out of range: End of sequence[[{{node IteratorGetNext}}]]Predicted relaxation level: 3.55% | Actual outcome: NOT RELAXEDPredicted relaxation level: 36.90% | Actual outcome: RELAXEDPredicted relaxation level: 1.24% | Actual outcome: NOT RELAXEDPredicted relaxation level: 52.69% | Actual outcome: RELAXEDPredicted relaxation level: 0.01% | Actual outcome: NOT RELAXEDPredicted relaxation level: 0.44% | Actual outcome: NOT RELAXEDPredicted relaxation level: 35.82% | Actual outcome: RELAXEDPredicted relaxation level: 67.30% | Actual outcome: RELAXEDPredicted relaxation level: 61.27% | Actual outcome: RELAXEDPredicted relaxation level: 0.03% | Actual outcome: NOT RELAXEDPredicted relaxation level: 11.89% | Actual outcome: RELAXEDPredicted relaxation level: 84.09% | Actual outcome: RELAXEDPredicted relaxation level: 45.33% | Actual outcome: RELAXEDPredicted relaxation level: 83.76% | Actual outcome: RELAXEDPredicted relaxation level: 84.47% | Actual outcome: RELAXEDPredicted relaxation level: 0.11% | Actual outcome: NOT RELAXEDPredicted relaxation level: 47.62% | Actual outcome: RELAXEDPredicted relaxation level: 0.04% | Actual outcome: NOT RELAXEDPredicted relaxation level: 22.17% | Actual outcome: RELAXEDPredicted relaxation level: 1.19% | Actual outcome: NOT RELAXEDPredicted relaxation level: 77.79% | Actual outcome: RELAXEDPredicted relaxation level: 31.63% | Actual outcome: NOT RELAXEDPredicted relaxation level: 56.86% | Actual outcome: RELAXEDPredicted relaxation level: 4.79% | Actual outcome: NOT RELAXEDPredicted relaxation level: 84.44% | Actual outcome: RELAXEDPredicted relaxation level: 30.84% | Actual outcome: RELAXEDPredicted relaxation level: 0.37% | Actual outcome: NOT RELAXEDPredicted relaxation level: 81.39% | Actual outcome: RELAXEDPredicted relaxation level: 0.89% | Actual outcome: NOT RELAXEDPredicted relaxation level: 66.95% | Actual outcome: RELAXEDPredicted relaxation level: 40.20% | Actual outcome: NOT RELAXEDPredicted relaxation level: 84.41% | Actual outcome: RELAXEDPredicted relaxation level: 36.15% | Actual outcome: NOT RELAXEDPredicted relaxation level: 82.13% | Actual outcome: RELAXEDPredicted relaxation level: 85.40% | Actual outcome: RELAXEDPredicted relaxation level: 6.47% | Actual outcome: NOT RELAXEDPredicted relaxation level: 80.10% | Actual outcome: NOT RELAXEDPredicted relaxation level: 75.26% | Actual outcome: RELAXEDPredicted relaxation level: 82.98% | Actual outcome: RELAXEDPredicted relaxation level: 0.00% | Actual outcome: NOT RELAXEDPredicted relaxation level: 0.03% | Actual outcome: NOT RELAXEDPredicted relaxation level: 60.53% | Actual outcome: RELAXEDPredicted relaxation level: 21.79% | Actual outcome: RELAXEDPredicted relaxation level: 0.87% | Actual outcome: NOT RELAXEDPredicted relaxation level: 18.36% | Actual outcome: RELAXEDPredicted relaxation level: 69.78% | Actual outcome: RELAXEDPredicted relaxation level: 5.55% | Actual outcome: NOT RELAXEDPredicted relaxation level: 83.83% | Actual outcome: RELAXEDPredicted relaxation level: 11.75% | Actual outcome: NOT RELAXEDPredicted relaxation level: 39.90% | Actual outcome: RELAXED
39/50 Generally on the money with less loss. Again though it’s not easy to tell with this kind of data as it can be hard to keep the mind relaxed during the relaxed data gathering parts.
Source: Mind Patterning – Phase 1