Bugs in NXP Kinetis Ethernet Driver

The SDK^[1] drivers provided by NXP for use on the Kinetis K64 platform are extensive, well-tested and … not perfect. This article shows three bugs found in the ethernet driver. Note that none of this is original content; I merely put it together here for my future reference.

Forgetting to check for zero-length buffers

I have only seen this bug happen once in two years and have not found a way to reproduce it at will. So the analysis below may or may not be correct.

The symptom was that the firmware froze upon triggering the assertion in lwip/port/enet_ethernetif_kinetis.c:

“Buffer returned by ENET_GetRxFrame() doesn’t match any RX buffer descriptor”

After some Googling I found this forum thread, which suggests, in a roundabout way, that there is a missing check in fsl_enet.c. We have to add following to ENET_GetRxFrame():

if (curBuffDescrip->length == 0U)
{
    /* Set LAST bit manually to let following drop error frame
       operation drop this abnormal BD.
    */
    curBuffDescrip->control |= ENET_BUFFDESCRIPTOR_RX_LAST_MASK;
    result = kStatus_ENET_RxFrameError;
    break;
}

The NXP engineer on the forum explains: “I didn’t use this logic because I never meet this corner case and consider it a redundant operation.” I was curious if this “corner case” every happens, so I added a breakpoint, which got triggered after about two days of constant testing.

ChatGPT seems to think this check is necessary (but then again, I seem to be able to convince it of just about anything I do or do not believe in):

If you omit the check and DMA ever delivers a BD with length == 0: Your code will think it’s still in the middle of assembling a frame. It will not see the LAST bit yet, so it will happily advance to the next BD. That means the logic walks into an inconsistent state: rxBuffer may point to nothing, your rxFrame bookkeeping goes out of sync, and later you’ll crash on a buffer underrun, invalid pointer, or corrupted frame queue.

It remains to be seen if this check was behind my original crash, and if the body of the if statement is appropriate to handle the condition of unexpected zero-length buffer descriptor.

Credit: User pjanco first reported the error, while AbnerWang posted the solution. [source]

Incorrect memory deallocation

In fsl_enet.c, the function ENET_GetRxFrame() tries to deallocate the pointer of the receive buffer:

while (index-- != 0U)
{
    handle->rxBuffFree(base, &rxFrame->rxBuffArray[index].buffer,
        handle->userData, ringId);
}

First need to unpack some definitions to understand what the above means.

1. If we dig into the rxBuffFree() function, we discover it in the file lwip/port/enet_ethernetif_kinetis.c. The buffer to be deallocated is passed as a pointer void * buffer, and freed

   int idx = ((rx_buffer_t *)buffer) - ethernetif->RxDataBuff;
   ethernetif->RxPbufs[idx].buffer_used = false;
   
   

2. Next, what are rxFrame and rxBuffArray? The first one is of type enet_rx_frame_struct_t, which is defined in fsl_enet.h:

   typedef struct _enet_rx_frame_struct
   {
       enet_buffer_struct_t *rxBuffArray;
       ...
   } enet_rx_frame_struct_t;
   
   

   This allows us to see what is the type of rxBuffArray:

   typedef struct _enet_buffer_struct
   {
       void *buffer;
       uint16_t length;
   } enet_buffer_struct_t;
   
   

3. Finally, what is ethernetif->RxDataBuff? We find it declared in lwip/port/enet_ethernetif_kinetis.c as the static array in the function ethernetif0_init():

   SDK_ALIGN(static rx_buffer_t rxDataBuff_0[ENET_RXBUFF_NUM],
       FSL_ENET_BUFF_ALIGNMENT);
   ethernetif_0.RxDataBuff = &(rxDataBuff_0[0]);
   
   

   More precisely, RxDataBuff is a pointer to the first element of this array. This pointer therefore has the type rx_buffer_t*.

   That type itself is declared at the top of the same file as an aligned version of a uint8_t buffer:

   typedef uint8_t rx_buffer_t[SDK_SIZEALIGN(ENET_RXBUFF_SIZE,
       FSL_ENET_BUFF_ALIGNMENT)];
   
   

Now we can take a step back and think whether the idx calculation would be best done with the buffer itself, or a pointer to it. The calculation subtracts the following:

• rxFrame->rxBuffArray[index].buffer, of type void*, is a pointer to the memory location that stores the ethernet frame.

• ethernetif->RxDataBuff, of type rx_buffer_t*

The corrected code should pass the buffer pointer stored in .buffer, not the address of the .buffer field (omit the &):

handle->rxBuffFree(base, rxFrame->rxBuffArray[index].buffer,
    handle->userData, ringId);

Credit: This bug was found by KC on 7/31/2024.

Buffers not zero-initialized

Another bug in ethernetif0_init() in enet_ethernetif_kinetis.c: the ethernet buffer descriptor structs are declared static:

AT_NONCACHEABLE_SECTION_ALIGN(
    static enet_rx_bd_struct_t rxBuffDescrip_0[ENET_RXBD_NUM],
    FSL_ENET_BUFF_ALIGNMENT);
AT_NONCACHEABLE_SECTION_ALIGN(
    static enet_tx_bd_struct_t txBuffDescrip_0[ENET_TXBD_NUM],
    FSL_ENET_BUFF_ALIGNMENT);

The assumption is that since they are declared static, the descriptors will be zero-initialized at system startup. However, the macro AT_NONCACHEABLE_SECTION_ALIGN potentially places these descriptor in a special section that can bypass the zero-initialization, depending on the startup code and linker script.

In that case, we need to manually zero out these buffers. I put the following at the top of ethernetif_enet_init() in enet_ethernetif_kinetis.c:

// Buffer descriptors must be initialized to zero
memset(&ethernetif->RxBuffDescrip[0], 0x00, ENET_RXBD_NUM*sizeof(ethernetif->RxBuffDescrip[0]));
memset(&ethernetif->TxBuffDescrip[0], 0x00, ENET_TXBD_NUM*sizeof(ethernetif->TxBuffDescrip[0]));

Credit: This bug was also found by KC.

STM32MP135 Flashing via USB with STM32CubeProg

This is Part 2 in the series: Linux on STM32MP135. See other articles.

In the previous article, we built a Linux kernel and manually copied it to an SD card. This works for a first test, but quickly becomes annoying. Here, we show how to use the STM32CubeProg to flash the SD card without removing it from the evaluation board.

Tutorial

Note: You may find the extensive explanations in the Bootlin article about flashing a similar chip helpful.

1. Finish the build process as per the previous article, so as to have at least the following files under buildroot/output/images/:

   • tf-a-stm32mp135f-dk.stm32
   • fip.bin
   • u-boot-nodtb.bin
   • sdcard.img

2. Go to the ST website to download the STM32CubeProg. This unfortunately requires a registration and sign-up.

   Get the Linux version, unpack in a new directory, and run the installer (just follow its verbose prompts):

   $ cd cubeprog
   $ unzip ../stm32cubeprg-lin-v2-20-0.zip
   $ ./SetupSTM32CubeProgrammer-2.20.0.linux
   
   

3. Now plug in all three USB cables for the board. Set the DIP boot switches for serial boot (press in all the upper parts of the white rocker switches). Press the black reset button. If everything worked, you should be able to see the board under your USB devices:

   jk@Lutien:/var/www/articles$ lsusb
   ...
   Bus 001 Device 114: ID 0483:3753 STMicroelectronics STLINK-V3
   Bus 001 Device 012: ID 0483:df11 STMicroelectronics STM Device in DFU Mode
   ...
   
   

   The STLINK-V3 is what you can use to monitor the flashing progress via UART. Simply open a serial monitor:

   sudo picocom -b 115200 /dev/ttyACM0
   
   

4. Run the STM32CubeProg from the location that you installed it in to check that it is able to detect the board:

   $ sudo ~/cube/bin/STM32_Programmer_CLI -l usb
         -------------------------------------------------------------------
                           STM32CubeProgrammer v2.20.0
         -------------------------------------------------------------------
   
   =====  DFU Interface   =====
   
   Total number of available STM32 device in DFU mode: 1
   
     Device Index           : USB1
     USB Bus Number         : 001
     USB Address Number     : 002
     Product ID             : USB download gadget@Device ID /0x501, @Revision ID /0x1003, @Name /STM32MP135F Rev.Y,
     Serial number          : 002800423232511538303631
     Firmware version       : 0x0110
     Device ID              : 0x0501
   
   

5. If that worked, it’s time to prepare the images for flashing. Go to buildroot/output/images and create a file flash.tsv with the following contents:

   #Opt	Id	Name	Type	IP	Offset	Binary
   -	0x01	fsbl1-boot	Binary	none	0x0	tf-a-stm32mp135f-dk.stm32
   -	0x03	fip_boot	Binary		none	0x0		fip.bin
   -	0x03	ssbl-boot	Binary	none	0x0	u-boot-nodtb.bin
   P	0x10	sdcard	RawImage	mmc0		0x0	sdcard.img
   
   

   Finally, run the flashing command itself:

   sudo ~/cube/bin/STM32_Programmer_CLI -c port=usb1 -w flash.tsv
   
   

   The STM32CubeProg will go through the sequence of files you wrote into flash.tsv. First, the Arm Trusted Firmware (TF-A) gets written to the memory and executed. It then does some secure magic behind the scenes and accepts the next payload via the DFU protocol, the U-Boot. At last, U-Boot itself is executed and it in turn accepts the last payload: the SD card itself. Which was, after all, the only thing you wanted to transfer anyway …

Discussion

The tutorial above again presents the simplest method I have found so far, with a minimum of steps and prerequisites, to flash the SD card of the eval board without taking the card in and out. What’s the issue?

The STM32CubeProg comes in a 291M zip file, which gets installed as a 1.5G program. We use it to copy a disk image to the SD card. See the problem yet? Or let’s consider the on-board procedure: TF-A (4,212 files and 506,952 lines of code according to cloc) is used to run U-Boot (21,632 files and 3,419,116 lines of code), just so that a semi-standard USB DFU protocol can expose the SD card to write the image.

But why??? ChatGPT explains:

U-Boot became the standard since vendors upstreamed support there, and it offers cross-platform flashing via DFU/fastboot for factories and Windows users who can’t dd raw disks. It also doubles as the hook for A/B updates, rollback, and secure boot. In practice, this forces developers into a complex boot stack, even though most boards could just boot Linux directly from SD/eMMC and use a tiny DFU mass-storage tool for recovery.

A more likely explanation is that the boot process has acquired an unnecessary reputation for being difficult, so that few want to mess with it. If there is a working solution, it will get incorporated into the software stack, no matter how baroque. The warning has been around for a long time:

Big building-blocks […] can lead to more compact code and shorter development time. […] Less clear, however, is how to assess the loss of control and insight when the pile of system-supplied code gets so big that one no longer knows what’s going on underneath.

[… As] libraries, interfaces, and tools become more complicated, they become less understood and less controllable. When everything works, rich programming environments can be very productive, but when they fail, there is little recourse.^[1]

All these tool are intended to make our work easier, but as they are piled on without any reasonable limit, the resulting mess is ironically far more complicated than the problem they are solving. If the task at hand is to flash an SD card image, why doesn’t the firmware expose the medium as a USB mass storage device, so that standard tools like dd could be used to work with it? The cynical answer suggests itself … They didn’t know better.

Those who do not understand Unix are condemned to reinvent it, poorly.^[2]

Surely it cannot be too difficult to write a simple “bare-metal” program, which we could load to the board using the simple and well-documented UART protocol implemented in the ROM of the STM32MP1. The program would be very small and quick to load. The program would expose the available media as mass storage devices, and that’s it.

But … You may object, we need U-Boot anyways, otherwise how are we to load Linux? As we will explain in a future article, that is not so. U-Boot is entirely unnecessary for a large class of embedded Unix applications.

What Unix Contributed

Unix was built on a handful of ideas that turned out to be both powerful and practical. The following discussion blends established Unix facts with interpretive commentary; it does not claim to describe any single historical Unix precisely.

Programs and the Shell

The shell runs commands as programs. There’s no special class of built-ins; if you want a new command, you write a program. By default, programs read from standard input and write to standard output, unless redirected.

Most commands are small filters for text streams. They do one job, and they work together naturally. Connecting them with pipes lets you build bigger tools out of simpler ones.

The File System Abstraction

Everything is a file: user data, programs, directories, and even devices. Directories form a tree; each entry points to an inode, which knows where the data blocks live. Devices show up as files too.

This means that I/O and storage use the same calls: open, close, read, write. That’s the interface for everything. Executables and data files are stored in the same way, reinforcing the idea that a single abstraction suffices.

Processes and the Kernel

The kernel is deliberately small. It multiplexes I/O and leaves the rest to user programs. Even init, the first process, is just a program: it opens terminals, prints the login message, and starts shells in a loop.

Processes come from the fork/exec pair. One process copies itself, then overlays the copy with another program. The idea is simple, and it works.

System calls are invoked by a trap instruction, wrapped in library functions so programs don’t depend directly on kernel details. Programs stay independent, and the operating system can change underneath.

Small, Understandable, Portable

Unix was small enough that one person could understand the whole thing. That made it easier to modify, port, and teach. The manuals were short, consistent, and focused on usage, not internals. A second volume provided tutorials and background for those who wanted more.

The guiding principle was: be general, but not too general; portable, but not too portable. If you try to solve every problem in advance, you get bloat. By keeping it modest, Unix was more useful—and paradoxically more general and portable—than larger systems.

The 80/20 Rule

Some parts were machine-specific, usually device drivers or bits of assembly. But not many. Most code was reusable, and the exceptions were small. An array of function pointers mapped device numbers to driver routines; that was about as complex as it got. For example, a character device^[1] driver needs to expose the following functions:

extern struct cdevsw
{
	int	(*d_open)();
	int	(*d_close)();
	int	(*d_read)();
	int	(*d_write)();
	int	(*d_ioctl)();
	int	(*d_stop)();
	struct tty *d_ttys;
} cdevsw[];

The 80/20 rule applied everywhere: make most of the system simple and portable, accept a little complexity when it really pays off. Code was meant to be 80% reusable, not 100%, which avoided the kind of rigidity seen in later systems.

Self-Hosting and Accessible

Unix came with all its own sources and tools. It was self-hosting, and people could read, study, and change the code. The system included what you needed, and nothing more. No useless programs, no dead code, and very little irrelevant platform-specific clutter.

The philosophy was to write programs you would actually use, not ones meant to satisfy a standard or some hypothetical future need.

Simplicity Above All

The enduring lesson of Unix is that simplicity beats complexity. Interfaces were orthogonal, text was the universal medium, and programs were small and self-contained. Each one did one thing, and did it well.

That philosophy proved more important than any single feature. It made Unix portable, teachable, and durable. It showed that you don’t need a committee or a grand design to build something powerful. You need clarity, restraint, and the discipline to write only what you need.

Reflections and Extensions

Unix also suggests how to go further. Small, portable, self-contained programs can approach the kind of stability that TeX achieved—systems so refined that they don’t need to change.

Portability itself can be modular. The Wollongong group^[2] showed this by first porting Unix piece by piece to an Interdata 7/32, running it alongside the host system, and then replacing the host functions with assembly routines. That approach points toward kernels that are more modular, where pieces like fork and exec could be reused without bringing along a whole scheduler.

Device drivers can also be simplified. One idea is to treat them as user processes whose IDs match their device numbers. They would implement the usual open, read, and write interfaces, but otherwise behave like ordinary programs: start and stop freely, hold their own memory, receive signals. The kernel would not “manage” them, yet the familiar Unix file interface would still apply.

The same lesson holds today. Artificial intelligence can sometimes repair or adapt programs automatically, but only if the systems are small and self-contained. Large, tangled software offers no foothold. Unix worked because it avoided dead code, avoided over-abstraction, and made each interface simple enough to understand and replace.

Finally, Unix showed that the way forward can’t be too innovative. If “the way” is too radical, no one will follow it.^[3] The genius of Unix was that it was just radical enough.