Machine Learning-powered EV charger availability detection with Raspberry Pi Pico W

If you’ve ever experienced the panic of not being able to find an open spot for your vehicle at a charger, this Raspberry Pi-powered solution may be for you. This tutorial is courtesy of our good friend Sandeep Mistry, Principal Software Engineer at Arm. Sandeep is giving a talk about this case study at Embedded World in April — drop by if you’ll be at the event!

As electric vehicles (EV) gain further adoption, businesses are investing in infrastructure for employees to charge their vehicles at the workplace. However, this can lead to underuse and scheduling conflicts with the EV chargers. Many of my coworkers in the Arm office in Cambridge were switching to electric vehicles in 2023, including my colleague David Henry who mentioned these challenges while I was visiting the office last June. This sparked the idea to build a system that could inform people in the office of the EV chargers’ availability.

This blog will explore how computer vision and an embedded machine learning (ML) image classification model can be deployed to an Arm Cortex-M0+ based Raspberry Pi Pico W board to inform employees when an EV charger is occupied or available.

The system will run continuously and:

1. Capture an image from a camera module connected to the Raspberry Pi Pico W board.
2. Use the captured image as input to an image classification ML model that detects if a car is present or not present.
3. Smooth the ML image classification model’s output using an exponential smoothing algorithm.
4. Send a message to a Slack channel, using the Slack API, when the EV charger’s occupancy state changes (available to occupied, occupied to available).

To preserve privacy, all ML inferencing will be done on the board’s Arm Cortex-M0+ processor using the TensorFlow Lite for Microcontrollers (TFLM) library. The application running on the Pico W board will only send the occupied or available status of the EV charger over Wi-Fi to the Slack API.

The source code for this project, including the code to train the model and inferencing application, can be found on GitHub.

Camera input

In a previous guest blog, I’ve covered how to connect a Raspberry Pi Pico to a Himax HM01B0-based Arducam HM01B0 Monochrome QVGA SPI Camera Module for Raspberry Pi Pico and capture monochrome images in real time.

This project will use the same camera module but swap out the Raspberry Pi Pico board with a Pico W board, which includes built-in Wi-Fi connectivity. Images will be captured from the camera at a resolution of 160×120.

The camera module will be connected to the Pico W board as follows:

The HM01B0 Library for Pico will be used interact with the camera module from the Pico W board. The library includes a usb_camera example that demonstrates how to capture images in real time and stream to a PC using USB and the USB video class (UVC).

For more information, please read the Real-time monochrome camera input on Raspberry Pi Pico guest blog.

Sending messages to Slack

In a previous guest blog post, I covered how to use the Pico W board with the Slack API to send and receive messages.

For this application, we can re-use the pico-sdk based C code to send messages to Slack using the Slack API. Follow the “Configuring Slack” section of the Create your own Slack bot with a Raspberry Pi Pico W blog to configure a Bot Token to use in the application to send messages.

The Pico W board will send messages using the Slack API’s chat.postMessage API. An HTTPS request to send a message to a channel would be structured as follows:

POST /api/chat.postMessage HTTP/1.1
Host: slack.com
Content-Type: application/json;charset=utf8
Authorization: Bearer 
Content-Length: 

{"text": "", "channel": "name of Slack channel>"}

This portion of the application will leverage the pico-sdk with FreeRTOS kernel, lwIP, Mbed TLS, coreHTTP and cJSON.

Training the ML image classification model

Dataset

Every machine learning model begins with a dataset, as ML models need something to learn from.

For this application, we will use images from the CNRPark+EXT Patches dataset, which contains 150×150‑pixel images for parking spaces that are occupied or available. Images in this dataset are licensed under the Open Data Commons Open Database License (ODbL) v1.0 license.

The images in the dataset can be classified into two categories (classifications):

• No car
• Car

Model Architecture

TensorFlow Lite for Microcontrollers (TFLM) includes a person detection example that detects the presence or lack of presence of a person in a 96×96-pixel monochrome image. The ML model used in this example uses the MobileNet V1 architecture with an alpha of 0.25.

Since this is a well established model for image classification on microcontrollers, we can create a model with the same architecture, but trained on the CNRPark+EXT Patches dataset.

The following Python code can be used to create a model using TensorFlow:

import tensorflow as tf

mobilenet_025_96 = tf.keras.applications.mobilenet.MobileNet(
    input_shape=(96, 96,1),
    alpha=0.25,
    dropout=0.10,
    weights=None,
    pooling='avg',
    classes=2,
)

Training pipeline

A Jupyter Notebook will be used to train the custom MobileNet V1 model on the CNRPark+EXT Patches dataset. This notebook can be used with a GPU enabled Google Colab instance.

Here’s a high-level overview of the steps in the notebook:

Dataset processing

1. The first step is to download and extract the dataset from https://github.com/fabiocarrara/deep-parking/releases/download/archive/CNR-EXT-Patches-150×150.zip to a dataset folder.

2. The images in the dataset are then organized in the following folder structure (0/ is the folder for images with no car present, 1/ is the folder for images with a car present) based on the information in the train.txt, val.txt, and test.txt files:

• dataset/
  • train/
    • 0/
    • 1/
  • test/
    • 0/
    • 1/
  • val/
    • 0/
    • 1/

3. The images are loaded from train, val, and test folders using the TensorFlow tf.keras.applications.mobilenet.MobileNet(…) API. This stage also converts the images to grayscale and resizes them to 120×120 pixels. (Note: the model will eventually need the images to be 96×96 pixel, but this will be done after the next step.)

4. The images are augmented with a data augmentation layer to prevent overfitting during training. The training input will be:

• Randomly flipped horizontally
• Randomly rotated
• Randomly zoomed
• Adjust contrast randomly
• Adjust brightness randomly
• Add random noise to the image

5. After data augmentation, the 120×120 images will be resized to 96×96 pixels.

ML model creation

1. A MobileNet V1 model is created with the tf.keras.applications.mobilenet.MobileNet(…) API. This model has an input size of (96, 96, 1), alpha of 0.25, dropout of 0.1, and two output classes. It is then modified to be compatible with TFLM by replacing the Reshape layer with a Flatten layer.

2. The model can be compiled with the model.compile(…) API and trained with the train and val splits using the model.fit(…) API.

3. After training is complete, the test split can be used to evaluate the model with the model.evaluate(…) API.

ML model exporting

1. In order for the model to run on the board it must be converted to TensorFlow Lite format using the tf.lite.TFLiteConverter.from_keras_model(…) API. This API will convert the floating-point model using quantization, so that 8-bit integer inputs and calculations can be performed on the Pico W’s Arm Cortex-M0+ processor for efficient inferencing.

2. Once the model is converted to a quantized .tflite file, the xxd command can be used to convert it to a format that can be compiled into the Pico W application.

Model limitations

ML models can only be as good as the data they are trained on; in our case the dataset contains photos of parking spaces with cars and without cars. For non-car objects, like traffic cones or wheelbarrows, the model will likely not output the desired output class.

To improve this, the dataset the model is trained on will have to be expanded to include these objects in the appropriate class. We’ll leave it up to you to think about other cases like this and expand the dataset accordingly.

Final application

Application overview

The application on the Pico W will do the following when starting:

1. Initialize the USB stack.
2. Initialize the camera.
3. Initialize the image classification model.
4. Initialize the Wi-Fi stack and connect to Wi-Fi.
5. Initialize the Slack client.

The application will have two FreeRTOS tasks, one to handle the USB stack and another task for the main application.

The main application will continuously loop and perform the following:

1. Capture an image from the camera.
2. If a PC is connected over USB, send the image data using USB UVC.
3. Perform ML inferencing with the captured image.
4. Smooth the prediction from the ML model using exponential smoothing.
5. Send a message using the Slack API if the detection state has changed from occupied to available, or available to occupied.

Building the application

All the application code, including a pre-trained model, can be found on GitHub.

Set up your computer with the Raspberry Pi Pico SDK and required toolchains. See the Getting started with Raspberry Pi Pico guide for more information.

Section 2.1 of the guide can be used for all operating systems, followed by the operating system-specific section:

• Linux: Section 2.2
• macOS: Section 9.1
• Windows: Section 9.2

In a terminal window, set the PICO_SDK environment variable:

$ export PICO_SDK_PATH=/path/to/pico-sdk

At the time of writing this guide, the latest version of pico-sdk was v1.5.1 and used v0.15.0 of the tinyUSB library. tinyUSB v0.16.0 or later is needed to use the USB UVC capabilities of the application. You can manually update the tinyUSB version of the application with:

$ cd $PICO_SDK_PATH/lib/tinyusb
$ git fetch
$ git checkout 0.16.0

Now the source code for the application can be cloned from GitHub:

$ cd /path/to/git/projects
$ git clone --recurse-submodules https://github.com/ArmDeveloperEcosystem/ml-image-classification-example-for-pico-w.git

Change directories to where the example code was cloned:

$ cd ml-image-classification-example-for-pico-w/

Create a build directory and change directories to it:

$ mkdir build
$ cd build

Run cmake with your Wi-Fi SSID and password, plus your Slack bot token and channel, then run make to compile the application:

$ cmake .. 
    -DPICO_BOARD=pico_w 
    -DWIFI_SSID="" 
    -DWIFI_PASSWORD="" 
    -DSLACK_BOT_TOKEN="" 
    -DSLACK_CHANNEL=""

$ make

Testing the application

Copy the ml-image-classification-example-for-pico-w.uf2 file to the mounted Raspberry Pi Pico boot ROM disk:

$ cp -a 
    ml-image-classification-example-for-pico-w.uf2 
    /Volumes/RPI-RP2/.

Use a serial monitor application such as screen to view the USB serial output from the board, replacing /dev/cu.usbmodem1234563 with the path of your board:

$ screen /dev/cu.usbmodem1234563

If everything is configured correctly, the board will now connect to your Wi-Fi network and then to Slack. You can use an application like QuickTime or VLC to get a live preview of the camera view.

If you don’t have access to a fleet of electric vehicles or prefer not to test in the cold of the (Canadian) winter, a set of toy EV cars can be used to test the application.

In the video clip below, you can see the Pico W board can perform approximately 1 inference per second, and the exponential smoothing requires a few inferences to toggle states from available to occupied or occupied to available. This is perfect for our use case, as parked cars do not move fast.

In the composite image above, we see:

• Left: live view from the Pico W camera feed
• Top right: USB serial output from the Pico W
• Bottom right: the Slack channel the Pico W is sending messages to

Conclusion

This guide demonstrated how a custom image classification model could be trained on an open dataset to detect the presence or lack of presence of a car in an input image. The model was deployed to a Raspberry Pi Pico W board, which captured image data from a Himax HM01B0-based camera module and performed on-device ML inferencing on the board’s Cortex-M0+ processor. Once the presence of a car changes from detected or not detected, the application uses the board’s built-in Wi-Fi connectivity to notify users that the EV charger is occupied or available.

As a next step, you could customize the ML model with additional images for the EV charger use case. Or, alternatively, train a new model on an entirely new dataset for other image classification tasks that tie into availability messaging, like hot-desking in an office space, or road transport and safety.