Overhauling The ESP8266’s Flash Memory Handling

If you’ve ever corrupted a flash memory on a power failure, you’ll be glad to hear that the ESP8266 SDK implements a very secure and almost infallible read/write management for its flash memory. The catch: It’s also very wasteful. For a single memory block of stored data, three memory blocks of physical flash memory are occupied. [Peter Scargill] enlightens us with a better solution.

When the ESP8266 writes data to its external flash memory, for example during an OTA update, it can’t simply overwrite the block holding the currently running program — it needs to write that data to a second block. Once the write operation is complete, it must keep track of which block holds the current data. For this, the ESP8266 SDK employs a third block, in which it stores the pointer to the current block. However, besides the block pointer, that third block stores no useful data.

It’s a deliberately wasteful technique that’s extremely useful for bulletproof firmware updates, but for storing additional data in the flash memory, you’d want a more efficient method. [Peter] managed to accomplish the same data integrity by using only two blocks per stored block of data. His method adds an 8-byte version counter to each block: When a block is read, the version counters are compared to retrieve the current data – when data is written, the version counter allows you to determine which block is the older one and can be overwritten.

[Peter] initially placed the version counter at the very end of a block, so it would naturally be written after the rest of the block has been written successfully. Unfortunately, flash memory practically requires you to clear an entire block before new data can be written to the same, so [Peter’s] method would leave the version counter in an erased state during the write operation. Eventually, he placed the version counter at the very beginning of the block using a flash-specific trick: When writing the data, he fills the first four bytes with ones (0xFFFFFFFF). This coincides with the erased state of the flash memory, allowing him to go back to the first four bytes and write the version counter after the entire block has been written successfully. A runnable test implementation of [Peter’s] overhauled flash read/write method for the ESP8266 can be found on his blog. Still too much overhead? Let us know in the comments!

Filed under: Microcontrollers, misc hacks