How to freeze the unfreezable
An alternative way of updating a MicroPython device
Introduction
The scenario is pretty much typical. A software project/application has been created and now it is time to be deployed on multiple devices. In the IoT world, the devices are microcontrollers and our topic is focused on those that run MicroPython firmware (Pycom devices & simple ESP32/ESP8266).
A way forward would be to simply sequentially connect each device and upload the related files to the device using ampy or Pymakr (through VSCode or Atom). This will work. It will take some time, but eventually it will work… hoping that the procedure will not randomly fail due to file upload errors.
An alternative would be to embed the project into the firmware, generate a .bin file and flash it on the devices using esptool or pycom-fwtool. Again, this is sequential, though it greatly increases the transfer speed and the chances that a flash will be completed successfully. Moreover, a .bin file can be sent to others and be applied, which is far more robust than sending individual code files.
However, this will only work if the project contains exclusively Python code files. That is because the “embedding of the project into the firmware” happens by copying all the required files into the “frozen” folder of the firmware and rebuilding it, converting all the Python code into bytecode. More info on this can be found here and here.
What if the project contains not only Python files, but others as well? For instance, what if the device acts as a WiFi access point and it serves an HTML file?
This is what microfreezer tool tries to solve by suggesting a method for bypassing the Python-only restriction.
How to
Start by downloading the source code of the MicroPython firmware of the required device and follow any instructions for preparing its dependencies (ex. Pycom, Generic Micropython). In this post we will use the folder structure used by Pycom devices.
Having the source code of the MicroPython firmware, any Python source code added in pycom-micropython-sigfox/esp32/frozen will be compiled and added with the firmware. This folder which will be referred to as “frozen” from now on and includes multiple other folders with necessary code. The suggested folder for placing custom Python files is pycom-micropython-sigfox/esp32/frozen/Custom/.
Now, let’s consider having the following project:
|- .git
|- lib
| |- lib_a.py
| |- README.md
|- html
| |- index.html
| |- favicon.ico
|- main.py
|- boot.py
|- README
|- LICENSEFirst step before running microfreezer is to prepare the configuration file by defining the following keys:
directoriesKeptInFrozen: select “lib” folder to be added in thefrozenmodules to be used directly by the user code.excludeList: select “README.md”, “README”, “LICENSE”, “.git” files to be ignored.flashRootFolder: use/flashas the root folder where the user code is normally placed.enableZlibCompression: enable zlib compression for smaller package size.- the rest of the files will be converted into a compressed format, packed into the
frozenmodules and will be ready to be exported to the “flashRootFolder”
Put these into a configuration file named config.json:
{
"excludeList": ["README", "README.md", "LICENSE", ".git"],
"directoriesKeptInFrozen": ["lib"],
"enableZlibCompression": true,
"flashRootFolder": "/flash"
}Run microfreezer
Having the config.json at the same path as microfreezer, it is ready to be executed.
python3 microfreezer ~/projects/my_new_project ~/projects/my_new_project_packedThe output is split into two folders, Base and Custom, following the directory design of Pycom devices:
|- Base
| |- _main.py
|- Custom
| |- lib_a.py
| |- _todefrost
| | |- base64_0.py
| | |- base64_1.py
| | |- base64_2.py
| | |- base64_3.py
| | |- microwave.py
| | |- package_md5sum.pyBase/_main.py: the file is automatically called by the device before calling (user space) main.py.Custom/lib_a.py:lib_ais a module that was part of thedirectoriesKeptInFrozen. This means that it will be globally available as a frozen module.Custom/_todefrost: the folder contains 3 main file typesbase64_.py: Each such file is a Python source file with two variables:PATH: the path where it needs to be extracted, which is a concatenation offlashRootFolderfrom the configuration file and the relative path of the file in the project.DATA: the contents of the file converted to Base64 and compressed withzlib.
package_md5sum.py: an md5sum of all the base64 Python files in_todefrost.microwave.py: the script responsible of decompressing and converting theDATAof each base64 file and placing it to the destination folder defined byPATH.
Taking the html/index.html file as an example, its original contents were:
<!DOCTYPE html>
<title>Title</title>
BodyAnd it is now described in Custom/_todefrost/base64_0.py as:
PATH="/flash/html/index.html"
DATA=b'eF6zUUzJTy6pLEhVyCjJzbHjsinJLMlJtQsBkTb6EA6XU35KJRcAPWcOfg==\n'Use the generated files to build the new firmware
Copy both generated folders to pycom-micropython-sigfox/esp32/frozen and rebuild firmware. Now it is ready to be flashed.
During the boot of the device:
- _main.py checks if md5sum of the files in “_todefrost” path has changed.
- If yes, runs “microwave” module that reads each file in “_todefrost” and unpacks (“or defrosts”) each file to the desired destination.
- Stores the md5sum of new packages at “/” or “/flash” for future checks on boot.
Some final thoughts
The advantages are clear:
- There is a single .bin file with all the code needed.
- In case each device has previously generated any configuration file, this procedure will not override it.
The downside is that:
- User code takes up space from the firmware available size, which is not unlimited. Fortunately, the compression of the data minimizes this effect.
- There is a conceptual argument regarding whether these files belong there… it is always a balance between what you gain and what you lose.
Link
- microfreezer: https://github.com/insighio/microfreezer
