This is a condensed version of my talk about adding some image detection onto the Raspberry Pi. With it only being 20 minutes I had to skim through quite a bit but hopefully it worked.
I guess I am now an international speaker!
Original Blog Series: Here
Meetup: https://www.meetup.com/Meetup-Computer-Vision-Paris/events/kjllfqybckblb/
Code: Main, MobileNetV2Base, and PiCameraManager
Presentation: ImageDetection/ComputerVisionParis.pptx
Today, I am going to walk through my process of moving my Keras trained model from my desktop onto my Android phone.
There are certain processes that I want to run against my model while I am away from my computer that I can’t do from the web. These range from some data collection to some larger data processing tasks. For example, I don’t want my web site to scrape all the spreads every time someone goes to it.
After I created my deep learning model to predict NCAA basketball scores in Google Colab I decided I needed to deploy it a few more places. My first change was using TensorFlow.JS to deploy on the web. After that, I pulled the model down to my desktop and wrapped it in a larger .NET application. Today, I decided that I wanted to create an Android application that would allow me to have my model on my local device instead of calling out to the web. It will also allow me to run a more robust set of commands that I couldn’t do on the web.
Part of being a GDE allows me to get access to certain groups and people at Google. One of the groups that I am part of is the ML on Mobile group that works on TensorFlow Lite (link). We have had a few meetings and that was the final push I needed to carve out some time and do this project.
This library is fantastic. Since all of my stuff was done in TFv2 and Keras it would be a simple conversion and then learning the Java API calls.
First, I had to convert my Keras model to a TF Lite model. This was as easy as the following commands:
#Convert the model
converter = tf.lite.TFLiteConverter.from_keras_model(restored_model)
tflite_model = converter.convert()
#Save the TF Lite model.
with tf.io.gfile.GFile('model.tflite', 'wb') as f:
f.write(tflite_model)
Second, I had to add that model to my ‘Assets’ folder in Android Studio.
Third, within Android Studio, I had to add code to the Gradle file to ensure the model doesn’t get compressed and to add the TensorFlow Lite libraries.
android{
...
aaptOptions {
noCompress "tflite"
}
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite:2.2.0'
Now, I needed to actually create the code that will interact with my model.
The key class here is the Interpreter class. This is the class that takes in your model and runs all of the predictions. In my case, I fought this like crazy. It started with trying to figure out how to turn the stream I got from the Assets folder into a File object (I had to write it to local storage, fyi). Then, I had to figure out what the input and output objects were going to be.
I have a simple model in that it takes in 6 decimals and outputs a single number. So, my input was a simple float[] and my output was float[][]. Here is the code:
File mdl = CreateFile(); //Creates the TF Lite model if it doesn't exist
Interpreter intp = new Interpreter(mdl);
//Create a 6 element float array. NOTE: I needed to do some normalization.
float[] inputs = BuildInputArray( 72.1,63.8, -5.1, 65.1, 70.8, -2.3);
//Create the output array that returns a Tensor
float[][] out = new float[1][1];
//Run the prediction
intp.run(inputs, out);
//Close the model
intp.close();
//Get the results
float results = out[0][0];
Well, there are my simple steps to get your trained model onto a mobile device. In the future, I will be adding a bunch of features so that I can do all the work from my phone and not have to go to my desktop each day during the season to get my gambling picks.
Recently, Google DeepMind released a paper on their most recent RL agent called Agent57. It was named after the 57 Atari 2600 games released called the Arcade Learning environment. From this the Atari57 benchmark was created to summarize the ability of the agent. The idea was that these games were different enough that you would have to have an intelligent agent. It was a step closer to “General Intelligence” if someone was able to create an agent that would outpace the average human.
In 1989, Chris Watkins developed the Q-Learning algorithm (my Jupyter Notebook on it!). The next enhancement was the Double Q Learner (another notebook) in 2010 by Hado Van Hasselt that added a second Q table. DeepMind then added a neural network in 2015 and created a Deep Q-network (my notebook!). Much like before, a second network was added in 2016 the Double Q-network (I have a whole set of notebooks!!).
At this point, I stopped researching these and have been out of the loop. Once I figure out unlimited time I will have to get back into it. Luckily, the researchers did not stop. They added Prioritised Replay, Dueling Heads, and some distributions. Then they created R2D2 with better short term memory. 2019 came and Never Give Up was created with better exploration and memory. A major leap forward during this time was the use of distributed agents. This allows the agents to be scaled up and speed up learning.
To me, this is awesome that we were able to get an agent that could handle games with immediate rewards plus long term rewards. We were able to handle games with a large array of tasks as well as a smaller array of tasks. And, we were able to scale.
Does this mean we have found General Intelligence? No, absolutely not. It does, however, open up a wide variety of possibilities. It makes the unknown task possible to solve. We can deploy this agent into an environment and have a pretty solid expectation that it will be able to handle the task.
Anyway, this is awesome and I finally found some time to look into it. Maybe, I will even be able to break it out and play some games with it.
DeepMind Blog Post: Link
This is the FINAL entry in my 8 part series on adding some object detection to my Raspberry Pi.
Part 1: Introduction
Part 2: SD Card Setup
Part 3: Pi Install
Part 4: Software
Part 5: Raspberry Pi Camera
Part 6: Installing TensorFlow
Part 7: MobileNetV2
Part 8: Conclusion
It seems odd to have an “introduction” in my conclusion post but I like to keep the format the same (I copy the previous post to create this one!). I will just talk about some high level issues and some other topics and then wrap this whole thing up.
I guess I will start with the issues that I had during this build.
First, I don’t use SD cards very often so that caused some issues when the hardware bit caused my card to become read only. Had I known about this “feature” I would have been able to keep my 64GB card for future projects. It would have also allowed me to not get into the 32GB limit partition limitation.
Second, while I never thought about it I really should have known that the ARM processor would cause some issues. I guess my blind trust in PIP lead me to think it could do anything including ignoring the processor.
Thankfully, the python/TF community is awesome and someone did all the hard work for me and created a TF library that would run on the Pi.
My final issue was with the actual image detection. This was 100% on my and not the model. With my personal life getting crazier by the day thanks to a 3 year old I didn’t do all the research I would have liked. I really should have gone through the whitepaper and see what/how the model was trained. This would allow me to get a feel for what images I should expect to detect.
My immediate goal for this project was to use it to get into a few conferences. My first conference application was for KCDC and I have yet to hear back but I am assuming I won’t get in. My second was a repeat of last year with the Twin Cities Code Camp but their deadline was February 14th at midnight and I took that as the night of the 14th and not the morning so I was 8 hours late. I don’t know if I will try and get back out to Denver again this year or not.
I decided that if I don’t get into any conferences this year I will put that part of my life on hold. The travel was interesting but with a young family it wasn’t worth going so far away. Maybe in a few years I will try and pick it back up.
As far as the project goes, I would like to do some custom training and see if it could pick out members of my family or something along those lines. I do have FishButler in the Google Play Store that could use some fish detection.
Conclusion to by conclusion post! Turtles all the way down.
Code: https://github.com/ehennis/Blog/tree/master/ImageDetection
